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ABSTRACT 


Since the end of the Cold War, reduced budgets have limited technology growth 
in the defense industry making the use of Commercial Off-The-Shelf (COTS) software 
the accepted way to build systems. Twenty years ago, almost all DOD software-intensive 
systems were built by awarding large multimillion-dollar contracts to defense contractors 
to build systems from scratch. Consequently, with dwindling budgets, the military has 
recognized that they can no longer build an infrastructure independent of commercial 
industry. 

The use of commercial items does not reduce or eliminate the risks associated 
with the traditional development of software systems. Numerous programs have 
stumbled for the lack of careful consideration and identification of the unique risk factors 
imposed by commercial items. Even though the types of programs are diverse, there are 
common risk factors that can be identified from the past experiences of these programs. 

This thesis focuses on the critical risk factors and lessons learned associated with 
integrating commercial items into DOD software programs. It summarize lessons learn 
from programs that have made extensive use of commercial items, provides a risk 
checklist/questionnaire to assist PMs and developers in understanding the risks associated 
with their developments of a system using commercial items, and suggests mitigation 
strategies, which can be used as guidelines for the risk factors, to consider when adopting 
commercial components. Providing the starting point for a systematic structure approach 


to the risk management of commercial items. 
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I. INTRODUCTION 


A. AREA OF RESEARCH’ 


Building of systems from commercial items depends on successful evaluation and 
selection of the commercial software. A number of risks associated with commercial 
items have been identified in the literature. This thesis will research the risk factors 
associated with the use of commercial items in Department of Defense Management 
Information Systems (MIS); command and control systems; and weapons systems. This 
will provide valuable insight into reducing the risks associated with COTS-Based 


Systems (CBS) development. 


B. RESEARCH QUESTIONS 


Principal Research Question: What are the unique challenges and risk factors that 
need to be managed when selecting COTS products for Department of Defense Systems 
(DOD)? 


Secondary Research Questions: 


° What are the definitions of commercial items and risk management? 

e How are the familiar notions of risk and risk management affected by the 
presence of commercial software? 

e Is there a specific approach/process that addresses risk identification for 
commercial (COTS) based programs? 

° What makes the integration of commercial (COTS) products into a system 
different from traditional integration of items designed and produced for 
DOD? 


C. DISCUSSION 


Throughout the Department of Defense (DOD), operations and support costs are 
rising, with fewer dollars available for research, test and evaluation and procurement of 
new systems; thus, increasing the pressure to achieve more with less. To achieve this 
goal, DOD is expanding the use of commercial items to leverage the perceived massive 


technology investments of the private sector while allowing DOD to reap the benefits of 


1 


reduced cycle times, faster insertion of new technologies and lower life cycle costs, 
increasing system stability, higher number of alternative solutions and increasing level of 
system interoperability. These advantages also bring related disadvantages, including 
integration difficulties, performance constraints, and incompatibility among products for 
different vendors. 

The use of commercial items does not reduce or eliminate the risks associated 
with the traditional development of software systems. Despite the risks, if COTS 
acquisition is addressed correctly it can provide significant benefits for buying 
commercial software; unfortunately the DOD policy on the risk management of 
commercial items is lacking. In the new less-restricted DOD directive 5000.1, the 
program manager is expected to tailor risk management practices to the needs of the 
program. Tailoring DOD risk management policy to support commercial items leaves 
the program manager with too much guesswork. A program manager using commercial 
items cannot reasonably benefit from DOD risk management guidance, procedures, and 
tools because he or she is focused on new development program risks and risk 
management practices. Missing are any explicit considerations of unique commercial 
items risk and risk management. Underestimating the risks associated with commercial 
software has often resulted in longer schedule delay, higher development cost, and higher 
maintenance cost. 

Because of the numerous risks inherent in the use of commercial items, there 
needs to be a flexible and proactive commercial item based risk management approach to 
avoid common mistakes in commercial items utilizations. Government system 
integrators need to be aware of the differences between military and commercial 
acquisition and the potential challenges and risks that need to be managed. The objective 
of this thesis is to focus on the critical risk factors associated with integrating commercial 
items into DOD software programs. It will summarize lessons learn from programs that 
have made extensive use of commercial items and offer suggestions for reducing the risk 


of developing systems with commercial items. 


SCOPE OF THESIS 


The scope of the thesis will include the following: 


In-Depth review of available literature, DOD Regulations, Audits and 
Reports as they relate to the use of commercial items. 


Interview Program Managers (PMs) who utilize commercial products 
within their programs to reflect their experience and lessons learned from 
using COTS within their programs. 


Provide a questionnaire to serve as a checklist to identify and understand 
the risk factors associated with COTS for the PMs to complete. 


Summarize and analyze the questionnaires and various lessons learned 
found in technical documents about risk associated commercial items. 


Derive conclusion and provide suggestions for reducing the identified risk 
factors of commercial items. 


METHODOLOGY 


The methodology used in this research consists of the following steps: 


Conduct a literature search of books, journal articles, magazine articles, 
World Wide Web, and other library information resources regarding the 
definitions and history of commercial items and risk management. 


Analyze and evaluate lessons learned and audit reports to identify risk 
factors and challenges (reliability, maintainability, and availability) 
associated with COTS. 


Develop a questionnaire based on these factors and challenges. Send this 
questionnaire electronically and conduct interviews with Product 
Managers in the Department of Defense whose programs do or have tried 
to utilize commercial items 


Summarize and analyze results of the questionnaire. 


Propose mitigation suggestions for reducing these factors. 


- ORGANIZATION 


This thesis is organized into the following sections: 


Chapter II: Commercial Items Defined. The introduction of COTS 
products into system acquisition has resulted in new terms associated with 
this process. Chapter II provides the terms and definitions associated with 
commercial items. 


e Chapter III: Risk and Challenges of Commercial (COTS) items. Chapter 
III provides a summary of different sources, DOD technical reports, 
audits, and handbooks, used for identifying the unique challenges and 
risks within a system when using commercial items. 


e Chapter IV: COTS Risk Questionnaire/Checklist. Chapter IV provides 
the framework for evaluating and reducing the risk of software systems 
developed with commercial items. The key to successful development of 
any system is having a sound approach and asking the right questions. A 
critical set of questions were developed and sent to project managers to 
assist them with understanding the risks associated with the developed of 
systems using commercial items within DOD. 


e Chapter V: Questionnaire Implementation. Chapter V sending the 
questionnaire electronically to elicit responses and provide results on 
current DOD projects using commercial products. 


° Chapter VI: Mitigation Strategies and Suggestions. Chapter VI describes 
some mitigation strategies and offers some techniques/suggestions, which 
can be used as guidelines for the risk factors, to consider when adopting 
COTS components. 


e Chapter VII: Conclusion. Chapter VII provides thesis conclusion and 
recommendations. 


G. BENEFIT OF THE STUDY 


The practice of risk management does not benefit from “cookbook” solutions 
[STEV97]. Numerous programs have stumbled for the lack of careful consideration and 
identification of the unique risks factors imposed by commercial items. Even though the 
types of programs are diverse, there are common risk factors that can be identified from 
the past experiences of these programs. DOD PMs, developers, and contractors can 
benefit from these past experiences since they provide a starting point for a systematic 
structure approach to risk management of commercial items and assist them in 
overcoming these barriers within their programs. They will be able to identify, as early 
as possible, the common risk factors and lessons learned from similar programs and 
determine how to use these experiences to adjust strategies and manage the risks, thus 
enhancing their programs while meeting OSD goals for incorporating COTS into their 


programs. 


I. COMMERCIAL ITEMS DEFINED 


A. INTRODUCTION 


For the government, making greater use of commercial-off-the-shelf (COTS) 
products is becoming increasingly popular. Increased use of commercial products holds 
the hope of getting, at a reasonable cost, something that already performs the functions 
needed by government systems. Everyone from industry executives to Congress is 
suggesting that leveraging commercial capabilities will save time and money while 
improving the performance of software-intensive systems. To encourage this approach, 
then-Secretary of Defense William Perry directed in June 1994 that DOD acquisitions 
should make maximum use of performance specifications and commercial standards, thus 
increasing the opportunities to make use of commercial products. The purpose of this 
section is to explain the different ways that the term COTS has been used, and the role 
commercial products can play in a commercial (COTS)-based system. The following 
definition of COTS is provided for the sole purpose of understanding the intent and scope 


of this paper. It is not meant to be ‘the’ definition. 


B. DEFINING “COTS” 


The term COTS is a familiar and well-used term within industry and DOD. The 
term COTS is generally used to describe commercial products. It commonly refers to 
things that one can buy, ready-made, from some manufacturer’s “store shelf” (through a 
catalogue or from a price list). This usage, however, is imprecise and not universally 
accepted. The government, which needs a precise definition for procurement, has 
defined the term commercial item. On 26 June 2000, the Office of the Secretary of 
Defense (OSD) defined a commercial item in their report, “Commercial Item 
Acquisition: Considerations and Lessons Learn” [DODA00]. This official definition of 
the term commercial item is given in the Federal Acquisition Regulations (FARs), 


appendix A; a summary along with an illustrative example, figure 1, are provided here. 


A commercial item is 


(1) Any item, other then real property, that is of a type customarily used for 
nongovernmental purposes and has been sold, leased, or licensed, or offered for sale, 


lease or license to the general public; 


(2) Any item evolved from an item in (1) through advances in technology and 


is not yet available commercially but will be available in time to satisfy the requirement; 


(3) Any item that would satisfy (1) or (2) but for modifications customarily 
available in the commercial marketplace or minor modifications made to meet DOD 


requirements; 


(4) Any combination of items meeting (1), (2), (3) above or (5), below, that 


are customarily combined and sold in combination to the general public; 


(5) Services for installation, maintenance, repair, training, etc. if such services 
are procured for support of an item in (1), (2), (3) or (4) above, as offered to the public or 
provided by the same work force as supports the general public, or other services sold 


competitively in the commercial marketplace; 


(6) Services offered and sold completively in the commercial market-place at 


catalog prices; 


(7) Any item, combination of items or service referred to in (1) — (6), above, 
that have been transferred between or among separate divisions, subsidiaries, or affiliates 


of a contractor; 


(8) | A nondevelopmental item (NDI) developed exclusively at private expense 


and sold competitively to multiple state and local governments. 


The key point is that commercial product(s) are developed by a commercial entity 
for commercial purposes and for the general public. In order to gain the advantages of 
commercial products, commercial software products should be defined as those that are 
offered to the public and are actually used by the public in the same version as those in 


military applications. 


C, DEFINING NDI 


A closely related term often heard in government circles is 


“nondevelopmental item” (NDI). 
A nondevelopmental item is: 


(1) Any previously developed item used exclusively for government purposes 
by a federal, state, or local agency or government or by a foreign government that has a 


mutual defense agreement with the U:S.; 


(2) Any item described in (1) above that requires only minor modification or 


modifications normally available in the commercial marketplace to meet requirements; 


(3) Any item being produced that does not meet (1) or (2) above only because 


it is not yet in use. 


The key point here is that NDI refer to something already developed by someone 
else. It might have been developed by a commercial interest, but typically it will have 
been developed for some other government, department, or agency. Hence, what is 
commonly called “government off-the-shelf’ (GOTS) is a form of NDI item. A large- 
scale example of a NDI would be a fighter aircraft developed by some other nation. A 
more meaningful example in the current context would be radar developed for one 


aircraft that is available for use in another aircraft. 


D. COMMERCIAL ITEMS CLARRIFIED 


On January 5, 2001, Dr. Gansler, then Under Secretary of Defense for 
Acquisition, Technology and Logistics, issued a policy memorandum to clarify and to 
help overcome some of the barriers being experienced within the Department of Defense 
in utilizing commercial items. An Integrated Process Team (IPT) had been formed at his 
direction and was headed by both the Deputy Under Secretary of Defense for Acquisition 
Reform (DUSD (AR)) and the Director of Defense Procurement. The IPT was chartered 
to review DOD commercial item determinations and evaluate whether additional 
guidance, tools, or training were necessary. Dr. Gansler’s memorandum says that the IPT 


found “inconsistent commercial item determination and weak market research among the 
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obstacles that exist to broadening the use of commercial items within the DoD.” 


[DODA01] 


Dr. Gansler’s memorandum also provided clarifying definitions of FAR Part 12 


for greater consistency within DoD. Four of the most important of these are as follows: 


Commercial Off-the-Shelf (COTS): A product does not have to be commercial- 





off-the-shelf (COTS) to meet the “commercial item” definition. COTS items are a subset 
of commercial items. The commercial item definition is much broader than products that 
are presently available off-the-shelf. It includes items that have only been “offered” for 
sale, lease, or license to the general public, as well as those that have evolved from a 
commercial item and are offered for sale, even if not yet available in the commercial 
marketplace. However, evolved items must be available in the commercial marketplace 
in time to satisfy solicitation delivery requirements. In addition, all other elements of the 


commercial item definition at FAR 2.101 must also be met. 


Modified Commercial Items: When items available in the commercial market 
cannot meet the Department’s need, DoD must determine whether market items can be or 
have been modified so that FAR Part 12 can be used. Two types of modifications are 
available: (1) modifications of a type available in the commercial marketplace; and, (2) 
minor modifications of a type not customarily available in the commercial marketplace 
made to Federal Government requirements. For modifications of a type available in the 
commercial marketplace, the size or extent of modifications is unimportant. For minor 
modifications, the item must retain a predominance of nongovernmental functions or 


physical characteristics. 


‘ 


‘Of a Type”: The phrase “of a type” is not intended to allow the use of FAR Part 
12 to acquire sole-source, military unique items that are not closely related to items 
already in the marketplace. Instead, “of a type” broadens the commercial item definition 


so that qualifying items do not have to be identical to those in the commercial 
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marketplace. The best value offer in a competitive Part 12 solicitation can be an item that 
has previously satisfied the Government’s need but has not been sold, leased, licensed, 
nor offered for sale, lease or license to the general public (a nondevelopmental item as 
defined in 10 USC 403 (13). In this scenario, the phrase “of a type” allows the best value 
offer to qualify for a Part 12 contract as long as it is sufficiently like similar items that 
meet the government’s requirement and are sold, leased, licensed, or offered for sale, 
lease or license to the general public. In such instances, “of a type” broadens the statutory 
commercial item definition to allow Part 12 acquisition of a government-unique item that 
can compete with commercial items that meet the government’s requirement. This 
avoids the undesirable result of shutting out otherwise price-competitive preexisting 


suppliers of government-unique items from Part 12 solicitations. [DODA01] 





Government _Off-The-Shelf (GOTS) is also a commonly used term for 
nondevelopmental items (NDI) that are government-unique items in use by federal, state 
or other governmental agency or by a foreign government with which the United States 
has a mutual defense cooperation agreement. This area will also be excluded and this 


term will not be utilized in this thesis. 


Since COTS has been defined as a subset of “commercial items” and Dr. 
Gansler’s memorandum specifically addresses the broader scope of commercial items, 


this researcher will use the term “commercial item(s)” throughout this thesis. 





(1) (2) 

An item offered for sale, An item that evolved 
lease or license to the from (1) that will be 
general public available in time 


(3) 

Items that are minor or 
standard modifications 
of (1) & (2) 


(5) (4) 

Services procured for the > Any combination of (1), 

support of (1), (2), (3) & (4) (2), (3), or (5) customarily 
sold to the general public 


Commercial 
Item 
(6) (7) 
Services offered and sold Any of (1) thru (6) that have 
competively in the been transferred from 
commercial market- another of a contractor's 
place at catalog prices organizations 


(8) 

An item sold competitively in 
large quantities to local and 
state governments 





(1) 

Any previously developed 
item used by federal, state, 
local, or allied governments 


Non- ‘ 
developmental (2) 


Item (1) that require only minor 


modification 
Ny (3) 


Integration of NDI 
subsystems and components 





Figure 1. FAR Definition Summarized “From [.DODA96|” 
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E COMPARING COMMERCIAL (COTS) ITEMS AND NDI 


NDI and commercial items are similar in that they both already exist, which is 
what makes them attractive. They are different in that commercial products usually 
appear in some sort of catalogue or price list, whereas it may be more difficult to discover 
the existence of NDI. Commercial items used to be considered a subset of NDI, but now 
this position is reversed, since a restricted form of NDI qualifies as a commercial item 
(see item (8) under commercial item). In addition, support services such as installation 


and training are now also defined as commercial items. 


Because the specific emphasis within DoD is to increase the utilization of 
commercial items, and nondevelopmental items (NDI) are placed second in the priority 
of implementation, this thesis will only address commercial item usage. NDI literature 
will be reviewed inasmuch as problems experienced utilizing NDI, will for the most part, 
apply to commercial item usage as well. Also, there is a great deal of indiscriminate and 
imprecise terminology usage. Thus, when the term “NDI” is used, the literature may 
actually be referring to a commercial item. Thus, the same types of precautionary and 
risk-mitigation suggestion recommended in this thesis for commercial items may be used 


when acquiring nondevelopmental items. 


F. WHAT ARE COMMERCIAL (COTS)-BASED SYSTEMS (CBS)? 


A commercial (COTS)-based system is a system that has been built primarily by 
acquiring and assembling a set of components that are commercial products into a 
working system. These components may perform generic functions that are independent 
of the system’s application domain or services that other components of the system 


depend on. 


Incorporating commercial software products to form a commercial-based system 
involves designing their interfaces with other system components, and the work of 
integrating them. The integrator responsible for building the system does so by buying 
the components and assembling them into a complete system. This involves minimal 
code development as most of the components are not developed from source but are 


purchased off-the-shelf. The code development required is that necessary to: tailor the 
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commercial software; build the components that are not being supplied through 
commercial sources; and glue the commercial components together. The number of 
interfaces a commercial product has with other system components, and the number of 


components it interfaces with determines the extent of integration. 


One type of system is the commercial solution, a system in which a single 
commercial product or product suite, provided by one vendor, that may be tailored that 
provides the primary solution to the problem. The amount of tailoring and data 
conversion is often significant. These systems may be found in application areas with 
general concurrence on application practices, examples being personnel management and 


financial management applications. 


A commercial intensive system, commercial-aggregate system, one in which a 
number of commercial components have been acquired from different sources and are 
assembled into a complete system. Combining the functionality of the different 
components provides the system services; there is no dominant single commercial 
component. Often the use of the particular set of components is unprecedented and 
requires substantial resources to select and integrate a cohesive set of components. These 
systems may be found where the needs of the system cannot be satisfied by a single 
product or product suite or when the system’s operational procedures are unique. For 
example, a command and control system could be constructed from a client/server GUI 


system, a GIS system, a set of databases, and data analysis tools. 


G. RISK MANAGEMENT 


Risk Management, one of the most critical and most difficult aspects for project 
management, is a proactive approach for minimizing the uncertainty and potential loss 
associated with a project. It follows a two-stage, repeatable and iterative process of 
assessment (i.e., the identification, estimation and evaluation of the risks confronting a 
program) and management (i.e., the planning for, monitoring of, and controlling of the 
means to eliminate or reduce the likelihood or consequences of the risks discovered). It 


is performed continually over the life of a program, from initiation to retirement. 
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The unpredictable quality of commercial software creates a unique set of risks for 
software systems using commercial components. The CBS acquisition process, then, 
should include risk management, which identifies high-risk items that can jeopardize 
system quality and attempts to resolve them as early as possible to ensure rapid and 
successful delivery of the system. Early identification and understanding the risks 
associated with commercial items is the first step to ensuring that the acquiring activities 
can achieve the benefits of using commercial products. 

This thesis will focus on this first step of risk management, the assessment, 
identifying the unique problems, risk factors associated with integrating commercial 
items that have an important influence on whether a program will succeed or fail. It will 
provide the understanding of these unique risks to the program managers so they can 
successfully execute their programs and manage the risks associated with commercial 
items. It will also provide some suggestions for mitigating these factors. A wise 
manager will recognize risks that are specific to the integration of multiple products into 


a CBS and will take action to mitigate and control those risks. 


H. SUMMARY 


This chapter has explored the definition of commercial items and commercial 
based systems. It uses the broader term of commercial items from the FAR when 
referring to COTS. A CBS was defined as a system that has been built primarily by 
acquiring and assembling either a single component or a set of components that are 
commercial products and integrating them into a working system. In order for systems to 
be successful and achieve the benefits of using commercial items, they need to minimize 
the uncertainties and manage the unique risks associated with commercial items. With 
any risk, awareness of lessons learned by other organization that have implemented 
systems using commercial items will help build or strengthen strategies to address any 


unexpected challenges that may arise. 


The next chapter presents a review of literature and lessons learned from 
programs implementing and integrating commercial items in government programs. 


These sources identify a consistent pattern of unique risks associated with commercial 
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items. Project managers’ can capitalize on these many “lesson learned” to identify and 


understand the risks associated with commercial items. 
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ll. IDENTIFICATION OF CHALLENGES AND RISKS WITH 
COTS 


A. INTRODUCTION 


Commercial (COTS)-based acquisition strategy can be viewed as a risk 
management approach with the goal of reducing or eliminating the potentially severe 
risks and resultant adverse effects typical of custom-developed systems. However, while 
the use of commercial products can help to deal with these “custom acquisition” risks, 
using commercial products also introduces other forms of risk, stemming directly from 
the unique characteristics of commercial products. This increased use of commercial 
products by government organizations is creating a new acquisition operations and 
support environment which requires that a standard approach be established for 
identifying and managing (1.e., mitigating) the unique risks of commercial products. This 
chapter presents many of the unique risk factors associated with developing systems 
using commercial software. These factors are based on an extensive analysis and review 
of common government/industry lessons-learned as found in numerous technical 
documents, journals and other literature review. Summaries of some of the major 
documents used for the foundation of this thesis are provided in this chapter with the full 


list of the sources provided in the list of references. 


Examining the similarities and differences of organizations that have applied 
commercial (COTS) products, the successes and failures of those organizations, will 
allow us to identify a number of unique factors and significant capabilities that an 
organization must have to succeed with developing a system using commercial items. 


B. BACKGROUND INFORMATION 


1, United States Air Force Scientific Advisory Board: Ensuring 
Successful Implementation Of Commercial Items In Air Force Systems, 
April 2000 


The Air Force was frustrated over the lack of success of those programs 
attempting commercial products implementation and were concerned that the customers 


were expecting miracles. Just being told to “maximize use of commercial (COTS) items” 
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without guidance, training, infrastructure, processes, tools, metrics, incentives, and 
leadership won’t make it so. A panel was formed to look into a broad range of 
commercial hardware and software products involving varying degrees of integration and 
complexity; however, the commercial hardware considered was limited to computers and 
electronics. The primary purpose of this report was to capture the issues, pitfalls, myths, 
lessons learned, best practices and critical success factors associated with commercial 


(COTS) items. 


The panel made an assessment of 34 programs and organizations, table 1, 
covering three broad domains — management information systems (MIS); command, 
control, communications and intelligence (C3I); and weapon systems listing the well- 
recognized benefits and several not so well recognized risks to consider when utilizing 
commercial items. The panel observed about 25 common pitfalls that programs are 
experiencing with most struggling with the technology, processes and complexity issues 
of commercial items with a few failing miserably. Most of these pitfalls could have been 
avoided or mitigated if appropriate risk management processes or procedures were in 
place that people understood and followed. While the concept of a commercial (COTS)- 
based system is easily understood, the implementation is not. The successful 
implementation of commercial products impacts virtually every aspect of the acquisition 
process including acquisition strategy, source selection, program management, system 


development, integration, and sustainment. 


Of the 34 programs interviewed, five were considered exemplary and, generally, 
those with the most experience were realizing the biggest gains. The study panel 
identified the common characteristics between these five programs and _ strongly 
recommended that these critical success factors form the basis of an implementation 
policy within the Air Force (and DOD). To serve as a framework to drive acquisition 
strategy, source selection, program management and, indirectly, the aerospace industry. 
In addition, since everyone is on a steep learning curve, they recommended that a 
periodic or annual review be conducted to incorporate additional lessons learned into the 


policy until the situation stabilizes. 
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Program/Organization | Service Organization 
Advanced Amphibious Assault USMC General Dynamics Amphibious 
Vehicle Systems 
AF Operational Test and Evaluation | USAF AFOTEC/CNR 
Center 
AFPEO/LI for Logistics Info SPO, USAF AFPEO/LI 
Gunter, AFB 
AFRL COTS Initiatives USAF USAF AFRL COTS Initiatives USAF 
AFRL/MLM & /IFTA 

AWACS Computer Modernization USAF ESC/AWC 

B-2 Data Storage USAF ASC/YSA 

B-2 EFX 99 USAF Northrop Grumman 

Boldstroke, commonality initiative The Boeing Company 

Open Systems Architecture & 

Software Component Technology 

Bradley Fighting Vehicle USA United Defense LP 

CALCE Electronic Products and University of Maryland 

Systems Consortium 

DCAC/MRM - Define & Control The Boeing Company 

Airplane Configuration / 

Manufacturing Resource Mgt 

COTS Supplier Approaches DY 4 Systems 

Earth Sensor TRW Space & Technology 
Division 

F-117 & F-119 Engine Electronics USAF ASC/LPC & /LPR 

F-15E COTS-based Products & F-16 | USAF ASC & ASC/YPV 

Upgrade 

F-22 Program USAF ASC 

Global Broadcast System USAF Raytheon Systems 

GPS, Ground Control Segment USAF SMC/CZG 

GPS Receiver USAF TRW Space & Technology 
Division 

Ground Station TRW Space & Technology 
Division 

Reuse of COTS Software GTE Information Systems 
Division 

JASPO, Signal Intelligence USAF ASC/RAJ 

Infrastructure 

Joint Direct Attack Munitions USAF Program Director 

Large ADP Systems & Software TRW Federal Enterprise Solutions 

Development Process 

Manufacturing Resource Planning USAF MRP I Program Office 

COTS Implementation in the USAF ASC Commercial Aircraft 


Mobility SPO 








Program 
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Program/Organization | Service Organization 
New Attack Submarine and Acoustic | USN Lockheed Martin Undersea 
Rapid COTS Insertion Programs Systems 
Enabling E-Commerce & Distributed Interoperability Clearinghouse 
Computing 
Office of the Department of Defense | OSD ASD/C3I CIO 
Chief Information Officer 
PVS/EVS — Enterprise Visibility Boeing Information Systems 
Service 
Deputy Assistant Secretary for | USAF SAF/AQX 
Management Policy & Program 
Integration 
T-38C Avionics Upgrade Program USAF ASC/EN 
T-6A Joint Primary Aircraft Training | USN ASC/EN 
System USAF 
TacTech (Parts Management) Transition Analysis of Component 
Technology, Inc. 











Table l. 34 Programs or Organizations Reviewed “From [USAF00]” 


2. United States Air Force Space Command: Commercial Space 
Opportunity Study (CSOS), February 2000 


Caught between intensifying warfighter needs in space and tight constraints on its 
budget, the Air Force has been encouraged to explore options in the commercial market 
for enhancing space capabilities while reducing costs. Beginning in 1996, the Air Force 
initiated a series of these studies to determine the potential of commercial space to 
support Air Force space missions and requirements. The prior studies were general in 
nature, but identified promising opportunities in commercial space and advised Air Force 
leaders to move forward. In late 1997, the Chief of Staff, USAF, tasked the Chief 
Scientist of the Air Force to conduct a study called the “Doable Space” Quick Look 
Study. This study found that the military potential of commercial space was not well 
defined or understood, and recommended that the Air Force conduct “an aggressive study 
on exploiting the space commercial revolution.” The CSOS was chartered to build on 
and go beyond these previous studies and systematically exposed and assessed the 
technical, operational, policy and programmatic implications of potential commercial 


paths. 
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The CSOS set out to identify new opportunities to satisfy military requirements 
while reducing Air Force costs, expanding capabilities and/or achieving higher 
operational efficiencies focused on military activities that the commercial market was 
capable of implementing in both near and long term opportunities. The objective was to 
develop actions to address the issue of how the government and commercial space 
community can best work together to capitalize on commercial space opportunities and 
move the Air Force toward an integrated architecture that best serves the Air Force’s 
needs and budget. The core of the CSOS approach was to develop business cases 
through intensive discussions with industry, and to show whether “commercialization” 


would support or impair national security and readiness. 


The study’s approach was to look for areas where interests of the military space 
community and the commercial space community coincide. Proprietary discussions were 
held with interested commercial firms in areas of: customer satisfaction, market share, 
product development, industry growth potential, and cost control. By comparing the two 
sets of needs, the study identified common areas of interest to both communities. These 
common activities were launch, command and control (C2), remote sensing, 


communications, and navigation. The purpose of the discussions were threefold: 
1) To ensure that industry fully understood current Air Force space activities; 


2) To determine whether commercial providers could and would provide 


equivalent or superior services at costs lower than government costs; and 


3) To find out whether commercialization could be executed within the Air 


Force and not violate national policy requirements. 


Detailed business cases were solicited from those firms judged to have the 
capability to provide the space activity or function in question. The business cases 
described how the providers would meet Air Force requirements and why they thought 
they could profitably provide the services or products. All business cases were reviewed 
through several iterations until the study leadership felt it had a complete understanding 
of each provider’s concept and an understanding of the capability of the relevant 


industrial base as a whole. 


19 


3. Department of Defense Inspector General, Lessons Learned from 
Acquisitions of Modified Commercial Items and Nondevelopmental Items, 
Report No. 97-219. 23 September 1997 


The objective of this report was to determine lessons learned from the acquisition 
management of Defense systems developed and procured using modified commercial 
items and nondevelopmental acquisition strategies. They used available information 
from ongoing and past management efforts within DOD to identify many lessons learned 
from acquiring commercial items. They also reviewed audit reports addressing 
acquisitions of modified commercial and nondevelopmental items to determine whether 
the acquisition community is making progress in developing acquisition strategies that 
avoid some of the acquisition difficulties identified in earlier audit reports. The report 
summaries and categorizes the lesson learned from the buying organizations into critical 
program management elements and evaluates the effectiveness of their management 


controls. 


Specifically, they reviewed 37 programs, table 2, 10 Army, 23 Navy, and 4 Air 
Force acquisition programs in which the military department was acquiring modified 
commercial and nondevelopmental items as entire systems, subsystems, or major 
components. They identified 91 lessons learned in developing acquisition strategies for 
program definitions; program structures; program designs; contracting; program 
assessment; and decisions, reviews, and periodic reporting. In addition, they visited 22 of 
the 37 buying organizations to discuss specific lesson learned. For the most part, the 
organizations identified program uncertainties involved with acquisitions that affect 
product performance, quality, and logistical support. The report then identifies key 
acquisition strategies that would be disseminate to provide buying organizations with 
useful information on how to acquire modified commercial and nondevelopmental items 


from commercial suppliers. 





Department of the Army Modified Nondevelopmental 
Commercial 





Biological Integrated Detection x 
System 





Cargo Utility Commercial Vehicle 








xX 
Cargo Utility Global Positioning xX 
System Receiver 
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Department of the Army 


Modified 
Commercial 


Nondevelopmental 





Communications-Electronics 
Command Commercial 
Communications Technology Lab 


x 





Deployable Universal Combat 
Earthmover 





Lightweight Multiband Satellite 
Terminal 





Lightweight Video Reconnaissance 
System 





National Automotive Center 


Xx 





Near Term Digital Radio 


a a a 





Precision Lightweight Global 
Positioning System Receiver 


Xx 





DEPARTMENT OF THE 
NAVY 


Modified 
Commercial 


Nondevelopmental 





Advanced Deployable System 


x 





ARC-210 Very High Frequency/Ultra 
High Frequency Radio 


Xx 





Battle Group Passive Horizon 
Extension Surface Terminal 


x 





Combat Systems Engineering 


Xx 





Common Support Equipment 





Control Display Navigation Unit 





Fixed Distributed System 





Ground Proximity Warning System 


|< 





High Frequency Radio Group 





Hull, Mechanical and Electrical 
Equipment Data Resources System 


M/P<p><} [P<] P<] >< 





Joint Maritime Command Information 
System-A float 





Joint Power Projection/Real Time 
Support 





Medium Tactical Vehicle 
Remanufacturing 





Miniature Digital Assigned Multiple 
Access 





New Attack Submarine 





P100 Portable Firefighting Group 





Riverine Assault Craft 





Strategic Systems Program 


| >< | >< 








Submarine Message Buffer 





| >< 








21 









































Department of the Navy Modified Nondevelopmental 
Commercial 
Surface Ship Torpedo Defense 
a. Launched Expendable x 
Acoustic Device 
b. Multi-Sensor Torpedo xX 
Recognition and Alertment Processor 
Surveillance Towed Array Sensor x 
System 
Surveillance Towed Array Sensor xX 
System- Low Frequency Active 
Department of the Air Force Modified Nondevelopmental 
Commercial 
C-130J Aircraft XxX 
Commercial Aircraft Program x 
Military Products From Commercial Xx 
Lines 
T-1A Aircraft xX 














Table 2. 37 Modified Commercial and Nondevelopmental Programs Reviewed From 
“(DODI97]” 


4. FAA, COTS Risk Mitigation Guide: Practical Methods For Effective 
COTS Acquisition and Life Cycle Support, June 2002 


Since the introduction of the Federal Aviation Administration’s (FAA) 
Acquisition Management System (AMS) in 1996, the agency has fielded numerous 
commercial (COTS)-based systems into the National Airspace System (NAS). However, 
due the lack of any available internal or external guidance on how to manage the unique 
risks associated with commercial item acquisitions, the FAA as well as many other 
Government agencies has had a variety of experiences, many of them adding to system 
cost, schedule and performance risks. This guide was established to capitalize on these 
many “lessons-learned” from government and industry and imbed them in a practical 
manner within the context of an acquisition management process to more effectively 
acquire and provide life cycle support for commercial (COTS)-based systems. The guide 
provides the necessary underlying structured for a standardize approach to identify 


commercial (COTS) risks, analyzes the likelihood and consequences of the risks and 
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determine appropriate mitigation strategies to minimize their impact. It uses a set of 


worksheets and schedules to 


° Collect the information 

° Assess the risk level 

° Select the most suitable solution 

° Develop and deploy the solutions 

° Develop the technical rational to justify funding requirements 


Using the programmatic risk management element of it systems engineering 
progress (Figure 3.1), the FAA collected information on commonly experienced 
government and industry “lessons-learned” in the areas of reliability, maintainability, 
availability, supportability trends, and market research activities. They used the Internet, 
in-house experience, plus commercial and DOD publications and then converted them 
into risk factors. The risk factors were subsequently analyzed to determine practical risk 
mitigation strategies that could be included as part of early program acquisition, planning 
and which could be continuously applied throughout a system’s lifecycle. The 
information contained in this COTS risk mitigation guide would be incorporated as part 
of the overall risk management program for both existing and new start commercial 
(COTS)-based system acquisitions. Because technology and commercial products evolve 
rapidly, this process must be updated on a frequent (proactive) basis to avoid disruption 
(reactive) of system operations helping to avoid the disruptions caused by commercial 


items. 
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Programmatic Risk 
| om Management Planning 
Identify Risk 
Monitor and Track Risk 
Analyze Risk ae J 


Select Risk 
Mitigation Option 






Implement Risk 
Mitigation Plan 


Figure 2. FAA Programmatic Risk Management Process “From [FAAC02]” 


5. Department of Defense, Office of the Under Secretary of Defense for 
Acquisition and Technology, SD-2: Buying Commercial & 
Nondevelopmental Items: A Handbook, April 1996 


The Department of Defense must learn to use commercial and nondevelopmental 
items (NDI) effectively. Our ability to field affordable, state-of-the-art systems when they 
are needed, and to buy the millions of items needed to support our troops and fielded 
systems, depends on efficient use of available resources. The use of commercial items is 
no longer a question of “yes or no” but a question of to what degree. This handbook was 
developed as a guide for acquisition managers and personnel in other functional areas 
who are involved in buying commercial and nondevelopmental items (NDI). It is 
intended to help these individuals buy these items without inhibiting their use of creative 


and innovative strategies. 


It offers guidance on commercial and NDI acquisitions. It addresses the entire 
spectrum of acquisitions from systems, subsystems, assemblies, parts, and items of 
supply. This guide does not present a “cookbook” approach to commercial and NDI 
acquisitions; however, it does offer best practices and “lessons learned” along with 
additional “things to consider”, when buying commercial items. It presents case studies 


to demonstrate the effectiveness of using commercial products and practices in: 
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e Defining Requirements 


e Acquisition Process 
° Logistics Support 

° Test and Evaluation 
e Product Assurance 


With relatively short lead times for fielding commercial and nondevelopmental 
items, acquisition decision must not be made until tradeoff factors are identified, 
analyzed, and compared with other alternatives. In determining if use of a commercial or 
nondevelopmental item is feasible, personnel must tailor the guidance provided to the 
circumstances of their particular acquisitions and devote more program resources to 
addressing life-cycle support as more of the quantifiable program risk areas become 
known. 


6. Naval Sea Systems Command, Commercial Off-The-Shelf and Non- 
Developmental Items Handbook, March 2000 


This document was created in response to the challenge of DOD to use more 
commercial products in its military systems and the feedback from the Naval Sea 
Systems Command (NAVSEA) Commercial-Off-the-Shelf (COTS) Workshop held in 
Norfolk, VA in August 1998, where the Fleet and the NAVSEA user community 
expressed the need for NAVSEA guidance in the utilization of commercial items. Given 
the fiscal constraints under which NAVSEA operates, it has significantly increased 
Commercial Off The Shelf/Non-Developmental Item (COTS/NDI) in system 
acquisitions. Employing COTS/NDI is a prudent means of lowering the costs of 
acquiring equipment and systems that satisfy the Navy's needs. However, effective 
management of commercial hardware and software in Navy systems presents difficult 


and different challenges than traditional item acquisition and life cycle support. 


The document provides overall guidance and outlines approaches for developing 
successful acquisition, integration, and maintenance support strategies whenever 
commercial products are employed in military applications under the cognizance of the 
Naval Sea Systems Command. It is written from a global perspective and is intended to 
will help managers and implementers decide “what” factors to consider when employing, 


integrating and supporting COTS/NDI into systems; providing the framework to develop, 
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manage and execute a comprehensive, cost effective, COTS/NDI program based on DOD 
policy. By consolidating points from previously prepared plans, reports and studies, they 
leverage industry experience, lessons learned in other military applications, and current 
“best practices” to allow individuals to tailor these “what” factors to their specific 


program. 


It focuses on the acquisition and life cycle support of COTS/NDI for hardware 
and software for all NAVSEA programs (excluding Nuclear Propulsion Programs under 
the cognizance of SEA 08). It provides guidance for those disciplines involved in all 


phases of the COTS/NDI acquisition and life cycle support process: 


° Program Management 

e Technology Management 

° Systems Engineering 

e Test and Evaluation 

e Configuration Management 
° Logistics Support 

° Product Assurance 


While it is understood that every acquisition projects is unique and will vary 
greatly in complexity and requirements, the guiding tenets contained within this 
handbook should be reviewed, addressed and tailored accordingly to ensure successful 
application of COTS/NDI products to mission and program needs. The considerations 
contained in this guidance are not a “cookbook” for the application of COTS/NDI in 
NAVSEA Systems but intended to provoke questions that can then be answered by 
obtaining additional information on the COTS/NDI product. 


Ts Department of Defense, Data & Analysis Center for Software, 
Commercial-Off-The-Shelf (COTS): A Survey, December 2000 


The goal of this report was to survey the state of the practice in commercial 
(COTS)-based development and provide evidences of both successful use and failures of 
commercial (COTS) items in projects using them. Each commercial software component 
used is less code that needs to be designed and implemented by the developers. 


However, the developer is faced with the problem of ensuring that the commercial 
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product does perform the functionality that it claims to perform, that it does not 
intentionally perform functionality to be harmful to the system, that it will not adversely 
affect the system and that it can robustly respond to failures and anomalous inputs to 


prevent errors from propagating through the entire system. 


This report discusses the definition of commercial items and commercial (COTS)- 
based system, listing the pros, cons and issues in commercial (COTS)-based 
development. The use of commercial products in software development can require a 
considerable integration effort. Early estimation of this effort will help developers to 
choose the right commercial products and to decide whether to develop their own 


software instead of using a commercial item. 


The central part of this report is dedicated to survey methods and techniques that 
can be useful in commercial (COTS)-based development. There are little or no 
techniques that allow a user to assess the dependability of a commercial item (in the 
sense of availability of the functionality promised by the documentation, reliability of the 
functionality, availability and quality of documentation); therefore, successful 
implementation of commercial items depends on several factors. They try to summarize 
and analyze these factors by discussing how these factors influence the success of a 
project using commercial items. They then propose a process to support commercial 
(COTS)-based development with emerging standards and techniques for component 
integration discussed. By aggregating the factors they argue that using a dependable 
commercial item in a non-critical project based on a single commercial item is probably a 
reasonable choice. On the other hand, integrating several commercial products, just 
released, in a highly critical project means probably asking for trouble. In between lies a 
twilight zone where the decision on using commercial items has to be carefully evaluated 


case by case. 


8. A Management Guide to Software Maintenance in COTS-Based 
Systems, November 1998 


The use of commercial products can significantly change the process by which 
systems are maintained in their operational phase. We are just beginning to experience 


and understand these changes, and to recognize that life-cycle planning for commercial 
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(COTS)-based systems must take into account, in early planning, the issues that must be 
confronted in order to facilitate the maintenance phase. The objective of this guidebook 


was twofold: 


1) To provide planning guidance that results in low risk and cost-effective 
strategies for maintaining Commercial Off-the-Shelf (COTS) software 


products in commercial (COTS)-based systems, and 


2) To provide guidance on the preparation of a commercial (COTS) Software 


Life-Cycle Management Plan. 


The authors believe there is no single way to manage and sustain all commercial 
(COTS)-based systems; therefore, they developed a generic commercial (COTS) life- 
cycle plan and guidelines, describing the changes in the software maintenance process for 
systems using commercial products and how this process can be tailored for each system 
depending on its specific requirements and constraints. The basis for this guidance was a 
review of DOD and industry experiences and lessons learned in commercial (COTS) 
product applications, and attendance at meetings/workshops whose focus was on the 
implementation of commercial (COTS)-based systems. The guidebook considers the 
issues and risks in using commercial software over the life cycle and how to control 
them. Each commercial (COTS)-based system must look at and control the risks 
associated with the operational and technical characteristics of the system, the 
administrative policies and constraints placed on the program, and its financial situation. 
With the use of commercial products, the operation and maintenance phase starts sooner 
and continues for a much longer portion of a system’s life cycle. This makes early life 


cycle planning for maintenance of commercial products even more important. 


C. RISKS AND CHALLENGES 


Those who have followed the commercial (COTS) path have been learning the 
hard way that “just buying commercial items” does not necessarily lead to all of the 
desired benefits; problems and difficulties in acquisition procedures, product 
development, and logistics support, as well as the skills and experience required of 


personnel supporting the project are also introduced by the use of commercial products. 
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The extensive review of common government/industry lessons-learned and other 
literature identified the need for identifying and understanding the unique risks factors (or 
characteristics) and challenges associated with using commercial products. Risks that are 
not identified have the greatest chance of becoming problems, while risks that are 
identified and assessed have the greatest chance of being resolved. This section proposes 
a set of commercial items risk factors and challenges that can be classified into three 
categories of commercial risks with each category being further divided into elements or 
attributes. These categories are based on the numerous technical documents and personal 
experience. These categories are: 


e Process: The key considerations for developing and executing a 
successful acquisition process with the system/program requirements 
driving the organization to consider a commercial solution and the “fit” of 
those requirements with available commercial application package(s). 
Key areas are organizational, planning, tracking, contractual parameters, 
and evaluation of vendors’ experience and past performance. 


° Technology: The technical “fit” of the commercial product(s) with the 
existing and planned technical architecture, which supports an 
organization. How well the selected product will perform in the 
environment provided by the system. This includes the organization’s 
inherent technical challenges, such as the number and complexity of 
interfaces. 


° Implementation/Logistics Support: The process contains intermediate 
and final work product characteristics for the delivery of a commercial 
solution within an organization that includes - but is not limited to 
performance measures, vendors availability of support, testing and 
managing organizational change. 


1. Process Risk Factor: Commercial Standards 


Military equipment is required to operate under conditions not always required of 
commercial equipment. For instance: gunfire vibration, hot and cold extremes, and 
nuclear hardness are normal operating environments for military equipment. Commercial 
elements must be selected with these requirements in mind and there may not always be a 


commercial element that will work. 


Until recently the government drove technology development for military 
applications with large infusions of research and development (R&D) funding for 


custom-developed systems. The government could afford to specify exactly what was 
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desired, set requirements, and therefore promoted a “buyers” market of firms interested in 
meeting this demand. However, commercial products are typically designed and built to 
a variety of commercial standards that provide high-level guidance on such product 
characteristics as performance, quality and inter-operability fostering a “sellers” 
marketplace that is no longer driven by government R&D but by a much larger (and more 
profitable) commercial customer-base. This means that the commercial vendors have 
several customers and their products are manufactured to meet more general consumer 
demands, instead of being configured to meet specific and often-inflexible government 


requirements. 


This competitive environment and the rapid advances in the underlying 
technologies both drive and allow commercial product manufacturers to anticipate 
customer demands and to quickly develop and market their commercial products. 
Leaving the government with no control and minimal influence over how the commercial 
product evolves. Hence, if operational requirements are viewed as not negotiable, and 
the suppliers are unwilling to modify their commercial products to meet a unique military 
need, then the probability of finding an exact match between requirement and 


commercial product is diminished. [USAF00] 


2. Process Risk Factor: License Agreements 


Licensing is the vehicle for securing the use of products that you need; data rights 
and warranties are marketplace vehicles for protecting you (and the vendors) in the long- 
term use of those products. Understanding, mastering, and negotiating the licensing 
agreements with the vendor can have a tremendous impact on the success of your 
program. The fee structures of licenses and maintenance services may change without 
warning potentially resulting in a large cost impact. Different licensing and maintenance 
support options are available and negotiable which are sometimes unknown to customers. 
Enterprise licensing is rarely available and not many users within the DOD are using the 


same commercial software for the same purpose. 
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3. Process Risk Factor: Vendors Past Performance 


While many individual commercial products from different manufacturers might 
satisfy a particular set of functional requirements, there can be marked differences from 
one product to the next. Differences in the components manufacturers choose to use, 
quality assurance practices, manufacturing processes, labor force composition, market 
share, product support, upward/downward compatibility, corporate longevity, etc. can all 
affect the quality and therefore desirability of the commercial products that are offered 
for sale. The “buyer beware” maxim applies when choosing among apparently similar 
products. A vendor with a limited product line is likely to sacrifice a product to 
compensate for adverse market financial flux, while a vendor that employs ad-hoc 
development practices may not be able to sustain long-term product evolution, with other 
vendors offering little or no warning for produce releases/upgrades forcing the maintainer 


into reactive evolution mode to deal with obsolescence issues. 


4. Technology Risk Factor: Rapid and Asynchronous Changes 


Rapid turnover in the commercial product can be both a risk and a missed 
opportunity for the program manager who is unaware of these changes. If the sole 
objective of a system upgrade is to capture new technology more cheaply, then the use of 
commercial products may suffice. But many DOD systems have a 30- to 50-year lifetime 
or more, while the average commercial component is upgraded every 6 to 12 months and 
new technology emerges about every 18 to 24 months. Changes to a commercial product 
is driven primarily by the vendors’ perceptions of how to achieve a greater market share, 
how to anticipate customer demands and to quickly develop and market their commercial 
product to meet these demands. Thus the changes are based on what will sell well to the 
largest number of current and potential users, not on the unique needs of your particular 
programs. Vendors can add or take away functionality and may not place the same 
priority as you do on a change that you need or the retention of a feature on which you 
rely. The money that is saved by using a commercial product with proprietary interfaces 
can quickly be lost in maintenance as products and their interfaces change with the 
marketplace forcing customers to accept the upgrades of the new product in order to 


obtain the desired functionality. Even if the expected lifetime of a system is only five to 
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ten years, the fluctuations in commercial products and technology result in a state of 


constant change for any system employing them. 


Program management generally cannot control the frequency or the content of 


new commercial releases. Vendors are continually producing new products as well as 


revising the products that are already on the market with the timing of a new commercial 


product release tending to be asynchronous and independent of the new releases of other 


commercial products and components in the system. Upgrading to the latest version can 


result in risks such as the following and stresses the need to fully test each upgrade before 


incorporating it into the system: 


5. 


The new software version is incompatible with other commercial software 
products in the system, necessitating updating of those products too. 


The new version has new data formats that require changes to be made to 
the formats and contents of existing files and databases that were created 
by prior versions of the commercial software. 


The new version of the software is incompatible with the version of the 
hardware that is in the system. 


A new version of the hardware is introduced into the system that forces 
changes to the existing versions of the software to make them compatible 
or because timing has changed under the new hardware. 


The new version of the software changes the user interface in ways that 
require retraining operators. 


Changes in the consumption of time or memory resources by upgrades to 
commercial software are not compatible with the system requirements or 
the hardware capacities. 


The new version forces changes in the operational capabilities of the 
system because it no longer supports those capabilities in the same way or 
at all. 


A new version may provide more capabilities that may have to be 
suppressed or restricted due to security concerns. 


Technology Risk Factor: Integration 


The complexity of commercial software interfaces (e.g. operating system) with 


other commercial software products/applications, middleware, glue code, custom/legacy 


interfaces and integrating these multiple commercial products within one single system 


can lead to many interoperability problems: 
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e Lack of commonality with other products. It is possible (in fact, likely) 
that using a closed commercial product commits the user to proprietary 
interfaces and solutions that are not common with any other product, 
component, or system. 


e Performance feature clash. Commercial software vendors typically 
overload their systems and includes more features and functionality than 
are normally needed. Precautions must be taken during system 
development and subsequent upgrades to assure that these unused features 
do not clash with other software products. System’s architects must 
provide a way of masking the unwanted functionality so that it is 
inaccessible to the end-users and system programmers. 


e Multiple configurations. Changing generations of commercial products 
will occur. Depending on system complexity, the number of systems to be 
fielded and the length of time it takes to deploy them, the number of 
configurations could be significant. It is not uncommon for part numbers 
and software release identifiers to be the same but have different features 
or contents. For example, one production lot can be functionally 
equivalent to the next lot but contain different components and 
subassemblies. If a product contains firmware or if it is a software 
product, revisions can be made to subsequent product releases to correct 
deficiencies or to add unique features to enhance product marketability. A 
commercial product manufacturer may or may not elect to identify these 
configuration changes to its customers. 


6. Technology Risk Factor: Reliability 


Today, with the rush to bring many products to market, commercial products are 
notoriously error prone. One must recognize that a new product in a hot market segment 
will have problems, some potentially crippling to a system's reliability. This is a major 
concern since all military equipment must be highly reliable in the field. This sometimes 
requires equipment with failure rates not achievable with available commercial elements, 
forcing us to have an understanding about the vendor’s track record in building reliable 
commercial products. For certain applications, occasional errors and downtime may be 
acceptable. For other applications, the requirements may specify a Mean Time Between 
Failures and Mean Time To Repair that are very demanding-resulting in higher project 


cost. 


Evaluating commercial product reliability is somewhat different than evaluating 


the reliability of new development products. With commercial items the basic product is 
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already designed and its reliability established; however, detailed engineering and 
manufacturing data for commercial products is frequently not available. The 
Government is not involved in the design process and production testing for a 
commercial product. So the Government cannot continuously evaluate the reliability 


during design reviews, through analysis, or based on production test results. 


Consequently, the reliability assessment should be an operational assessment of 
the military application in the expected military environments since the buyer cannot 
control the basic design of a commercial item. The commercial product must pass the 
same reliability evaluations as the host components; otherwise the commercial product 
will be the weakest link in the chain of components and will be the determinant of 
software system reliability. The essential reliability analysis/tasks that must be 
performed are reliability predictions, system level Failure Mode Analysis, Failure 
Reporting and Tracking Analysis, and reliability verification. Consider the following 
when evaluating and fielding commercial products: 

e Reliability predictions may be difficult to obtain from the vendor. 


e Lack of data may limit the depth of failure mode analysis that can be done 
on commercial products. 


e Vendor’s definition of reliability data may be different than 
DOD/Government standards (e.g., Ao, MTBF, MTTR). 


Te Technology Risk Factor: Information Security 


When the government develops its own custom systems, it can specify and 
develop system security characteristics very precisely. Although vendors provide 
products with built in security features that address the commercial components 
interoperability issues, these products are typically developed to commercial standards 


with insecure defaults that introduces potentially significant security risks for several 


reasons. 
° The increased inter-operability among different products that meet 
commercial standards raises the chances that unauthorized access can be 

gained. 
e The use of commercial standards allows a greater number of people to be 


familiar with the software protocols used to manage information. This 
knowledge can be used to access or disrupt information flow. The “open- 
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ness” of a particular architecture, the degree to which it links with other 
external commercial (COTS)-based systems, and the nature of the security 
measures in place will determine the extent to which the products and 
systems using them are susceptible to unauthorized access. [FAAC02] 


Most commercial software is developed and implemented outside this country. It 
is common in software product development in the United States to use teams from other 
countries. In a government project with particular security requirements this presents 
major risk factors that may be unacceptable. Because a commercial product is essentially 
a black box, in the sense that the implementation of the software is most often hidden, 
leading to the possibility that a trap door or “Trojan Horse” may be embedded in the 
code, there may be a backdoor feature in the code, or unexpected capabilities. As a 
result, specific security relating to project requirements may not be guaranteed, as the 


security of the commercial products implementation cannot be ascertained. 


8. Implementation/Logistical Risk Factor: Product Obsolescence 
(discontinuation) 


Commercial products life cycles are generally much shorter with new versions of 
a commercial software package appearing as frequently as every 18 months. As 
succeeding generations of commercial products are introduced into the commercial 
market, the manufacturer must determine at what point when it is no longer profitable or 
desirable to support the older generation products. The manufacturer must make a 
tradeoff between selling its newer product line while at the same time not alienating the 
older generation product consumer base. After three or four upgrades, the manufacturer 
may choose to no longer maintain the earlier version incorporated in the military system. 
Also, a commercial product may be selected for a particular niche feature. If it turns out 
that the commercial market is not interested in this capability, there could be a lack of 


support or subsequent revisions that may exclude the feature entirely. 


When a commercial product is projected to be nearing end-of-life (EOL) (i.e., out 
of production) or end-of-service (EOS) (i.e., no longer supported by the manufacturer), 
the effects of these projected changes of state on the product and on systems using the 
product must be examined to determine what action if any is needed. It is not a foregone 


conclusion that all products declared to be EOL or EOS need to be replaced immediately 
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by newer versions of those products. Effects can range from no impact to high impact. 
The obsolescence support options that are available to address these impacts can range 
from taking no action to making a major system redesign [FAACO02]. The categories of 
impacts due to obsolescence are defined as follows: 


° No impact — In this case the product’s projected End of Life/End of 
Service status has no impact on the product or on any system using that 
product and therefore requires no action. The commercial product is 
considered reliable and there are sufficient spares (at acceptable prices, 
within the market or on-hand) to support the projected failure-driven 
demand over a pre-determined timeframe. 


e Low impact — This situation typically requires compatibility testing for the 
new product and a documentation change to identify the new product as a 
suitable alternative replacement part upon failure of the old part. The 
manufacturer’s next generation product is compatible  Le., 
interchangeable); if there are other manufacturer products that are 
compatible; and if there are no associated changes to interfacing products 
within the system. 


e Medium impact — This category of impact, like the low impact category, 
also applies when a commercial product must eventually be replaced. The 
manufacturer’s next generation product requires minor software changes 
and/or if related changes to interfacing products are required. 


° High impact — This category of impact, like both the low and medium 
impact categories, applies when a commercial product must eventually be 
replaced. However, a major impact situation exists if there are no 
compatible replacement products or technologies available on the market. 
This situation typically calls for a major redesign or an integrated system 
change. 


9. Implementation/Logistical Risk Factor: Proprietary Data 


A major drawback of including commercial items in a software system is the lack 
of visibility into how the commercial components were developed and an incomplete 
understanding of the components’ behavioral properties [SCHNOO]. A commercial 
product manufacturer remains in business because it owns and controls the research and 
manufacturing processes needed to meet market demands and to keep product costs 
competitive. As consumers, we have little insight into the specifics of how a commercial 
product has been developed and at times even into the details of how it behaves and why. 


This lack of visibility can hamper efforts to integrate the product with others to create a 
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larger system since the vendor may not be willing to provide detailed interface design 
documentation. As a result, the commercial product must be viewed as a “black box” 
with defined interface and performance characteristics but allowing no insight into the 
internal composition of that product. Because we do not have access to the source code, 
developers cannot modify the code to change the functionality of the commercial product 


(perhaps this is a good thing!). 


10. Implementation/Logistical Risk Factor: Underestimated Costs 


Accelerating the introduction of commercial products into government and 
military systems has been advertised as a “faster, better, cheaper” way of meeting 
requirements. However, unless a risk management program includes proactive mitigation 
strategies looking at the total ownership of cost for commercial products, the initial cost 
benefits can be offset by the often more costly fixes of the risks that weren’t effectively 
managed. Examples of the cost considerations for a commercial (COTS)-based 
acquisition strategy that need to be included as part of a total cost of ownership analysis 
include [FAAC02]: 


° Inadequate-planning costs — Probably the major life cycle cost-driver 
associated with the use of commercial products is the lack of effective 
planning and budgeting. When a program fails to apply commercial risk 
mitigation strategies, the program then loses the advantage of proactive 
planning and becomes increasingly reactive to emerging commercial- 
driven obsolescence situations. 


e Test and integration costs —Programs often underestimate the impact of 
testing commercial items. In addition to the actual costs of the test 
facilities needed to support the possibility of multiple system 
configurations, different commercial products with varying characteristics 
typically require that “glue code” be developed to allow the products to 
interact effectively. Each product must be tested for compliance to 
performance requirements, conformance to open system standards and 
compatibility with the system with which it will be integrated. 


° Modification costs — In some cases a commercial product must be 
modified to meet a particular or unique requirement. There is a cost to 
actually modify the commercial product itself. There is also a cost to 
assume life cycle management responsibility for that specific product 
because modifying a commercial product typically voids (unless functions 
are incorporated as part of the commercial product line) any warranty and 
the vendor will no longer provide support. This forces the life cycle 
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11. 


support for that product to be the responsibility of the acquiring activity. 
Costs for documentation, maintenance, training and spares costs will 
increase in this situation and must be planned for in the life cycle 
budgeting for that modified product. In addition, regression testing at the 
system level may be needed to ensure that the modification does not 
change the expected performance of the system. 


Configuration management costs — A consequence of using rapidly 
changing commercial products within a given system is the strong 
likelihood that an acquisition of multiple copies of that system will include 
more than one configuration of the commercial products used in the 
system. This situation not only demands a rigorous application of 
configuration management (CM) processes to document and manage 
system baselines but also requires that test facilities can replicate all 
fielded configuration baselines. 


Continuous system engineering costs —Commercial products forces a 
continuous system engineering effort, which adds additional cost to a 
program. Because commercial (COTS)-based systems are dynamic in 
nature, continuous systems engineering activities are needed to perform 
market  surveillance/research/investigation; analyze obsolescence 
projections; determine the available options to limit obsolescence impacts; 
and integrate the resulting information with new requirements and field 
data as part of the overall integrated program planning. You need to be 
able to go out and see what is going on in the marketplace and do some 
technology forecasting. 


Obsolescence management costs — There are costs associated with vendors 
or suppliers either dropping support for a commercial item or going out of 
business. The continuous system engineering activities needed to manage 
obsolescence can result in more frequent engineering changes to the 
system. The development, deployment and configuration management of 
these changes is an added cost that must be included in all commercial 
(COTS)-based system program planning. These costs must continuously 
be refined as system product obsolescence information is gathered and 
analyzed. 


Implementation/Logistical Risk Factor: Testing 


Commercial item’s capabilities may not always be as stated and demand 


excessive testing. There may be “hidden behaviors” associated with the commercial 


item, or bugs that affect the system. System integration and system level testing 


becoming even more vital, particularly in commercial (COTS)-based systems with many 


components (commercial (COTS), NDI, custom, legacy) where interoperability issues 


abound; therefore, one must ensure that the system fulfills all specified requirements and 
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that catastrophic faults are detected early. When commercial products are modified, the 
system may exhibit behavior different from the baseline requiring additional testing. In 
the past, unique custom designs were static so that the system tested was the system to be 
manufactured and deployed. Because of the volatility of systems incorporating 
commercial products, the system that is subjected to initial operational, testing and 
evaluation (IOT&E) is likely to be different from the system that enters production, since 
upgrades will have been incorporated. This constant evolution of a system is a cause of, 
since the consequences of the changes introduced into the initial production design are 


unknown. 


Testing and fault isolations are further complicated by the reality of restricted 
visibility into the behavior of the commercial product with any documentation that you 
may have, often incomplete and inconsistent, causing you to shift from a “white box” to 
“black box” testing process [SCHN00]. The integrator/testers must thoroughly test all 
inputs and outputs to “prove” to a vendor with potentially significant evidence that a 
particular commercial component is failing in a particular way, whether a detected failure 
is in a single component (and then, which one) or in the interactions among two or more 
components. This may take significant resources (time and very skilled technical staff) to 
isolate and resolve with the vendors. It is not the vendor’s responsibility for the ultimate 
success of your system; rather it becomes the integrating organization’s responsibility. 
You might want them to fix some bug or something, and they may or may not do it in 
that given release. Sit down in advance with your vendors to determine the routine for 
working out solutions to problems that will be encountered and find a way to 


cooperatively work together to find a satisfactory resolution. 


D. SUMMARY 


This chapter summarizes some of the major government/industry lessons-learned 
and other technical information to form a foundation for understanding and identifying 
the unique risk factors associated with developing systems using commercial software. It 
proposed a set of commercial item risk factors and challenges that were classified into 
three categories: process, technology, and implementation/support, with each category 


being further divided into specific elements or attributes. Only by understanding and 
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addressing the unique factors imposed by commercial items will program managers be 
able to attain their benefits and move towards market-oriented business practices that are 
better suited to the acquisition and life cycle support of commercial (COTS)-based 


systems. 
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IV. COTS RISK QUESTIONNAIRE/CHECKLIST 


A. INTRODUCTION 


Widespread use of commercial products in complex software systems poses many 
unique challenges and risks to both the developers and managers of systems using 
commercial items. It is virtually impossible to develop, modify or purchase commercial 
software without incurring risks. These risks can be known, unknown, or unknowable. 
Known risks are those that one or more project personnel are aware as concerns. The 
unknown risks are those that would surface (i.e., become known) if project personnel 
were given the right opportunity, cues, and information. The unknowable risks are those 
that, even in principle, none could foresee. Hence these risks, while potentially critical to 


project success, are beyond the purview of any risk identification method. 


Identifying the unique risk factors, assessing these factors, and controlling them 
are the keys to proper risk management with commercial items. Identification surfaces 
risks before they become problems and adversely affect a project. The sooner risks are 
identified, the better off the software managers, system engineers, project manager or 
decision-managers will be able to monitor, adequately mitigate, and resolve the risks; 
thus, achieving the desired benefits of using commercial items by ensuring a rapid and 


successful delivery of the system. 


B. IDENTIFICATION OF RISKS 


This section presents a questionnaire (Appendix B), checklist, for adopting and 
integrating commercial product(s) into systems or programs. The questionnaire is 
intended to provide a guideline (reminders), to any of the participants on the program, 
whether on the acquiring side or the contracting side, and focus their attention on the 
possible risk factors that individuals need to understand when using commercial items. It 
consists of two sections and is designed to provide a systematical tool, starting point, for 
managers to identify, investigate, and plan for the unique risks associated with 
implementing commercial items. It incorporates some of the most significant lessons 


learned from a variety of commercial implementations [ITRB99] and helps you evaluate 
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the risk by determining the severity of these risks for your own organization. While it 
may not be complete and has some obvious weaknesses, the checklist provides an 
organization, which is considering the adoption and integration of commercial items, 
insight into the areas that must be carefully considered. It also serves an important 
purpose for large-scale organizations by providing a repeatable approach for commercial 


items risk evaluation. 


1. Section I. Demographic Information. 


Collecting background information about the organization and experience of the 


individual(s), with commercial product(s), completing the questionnaire. 


2. Section II. Risk Questions 


A modification was done to the Information Technology Resources Board’s 
(TRB) “Risk Profile” [ITRB99], in order to structure 42 questions around the three broad 
categories that represents critical aspects required for the successful implementation of 
commercial items as identified in chapter three: process, technology, and 
implementation/logistics support, defined below, with several questions for each 
category. Even though some questions may not pertain to every project, these questions 
can be modified accordingly to meet the needs of the software project. Each question 
prompts you, the respondent, to think about key factors for a successful commercial 
product(s) implementation and how these factors pertain to your project within your own 
organization. 


° Process: The key considerations for developing and executing a 
successful acquisition process with the system/program requirements 
driving the organization to consider a commercial solution and the “fit” of 
those requirements with available commercial product(s). Key areas are 
organizational, planning, tracking, contractual parameters, and evaluation 
of vendor’s experience and past performance. 


° Technology: The technical “fit” of the commercial product(s) with the 
existing and planned technical architecture, which supports an 
organization. This includes the organization’s inherent technical 
challenges, such as the number and complexity of interfaces. 
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e Implementation/Logistics Support: The process contains intermediate 
and final work product characteristics for the delivery of a commercial 
solution within an organization that includes - but is not limited to 
performance measures, vendors availability of support, testing and 
managing organizational change. 


C. ASSESSING RESULTS 


Completing the questions and assessing/compiling the results should help 
managers better understand the significance level of risk associated with implementing a 
commercial product(s) and assist in identifying their causes given current business needs 
and organizational conditions. In turn, this knowledge will help guide the managers and 
let them take the steps necessary to minimize specific risks associated with the 
implementation of a commercial product(s) and formulate a strategy for acquiring 


commercial product(s) for their organization. 


1. Risk Severity Rating 


Answers to each question are provided by the choice a, b or c, which correlate to 
the three levels of risk: low, medium and high, respectively with points assigned for 
different levels. The level of risk is somewhat subjective and should be based on the 
experienced judgment of your best technical people with assigned responsibility. 


However, user input and feedback, along with industry comments also need to be 


considered. 

e Low risk, point value = 1, Actions within the scope of the planned 
program and normal management attention should result in maintaining an 
acceptable level of risk. 

° Moderate risk, point value =2, Corrective actions and/or careful 
monitoring of status by management are required to reduce risk or to see 
that the level of risk does not increase. 

e High risk, point value = 3, Significant corrective action and high priority 


management attention are required to achieve an acceptable level of risk. 
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2. Calculating the Risks 


To arrive at the programs total risk, each individual section of the risk categories 
must be examined. A box is provided for adding the total number of a, b, or c responses 
for each section. The table below illustrates this concept. The first column is the 
categories of the risks associated with a program implementing commercial product(s): 
process, technology, and implementation/logistics support. Moving to the right across 
the other columns in the table you will find space for recording the number of a, b, and c 
responses associated with each risk category. These individual point values are then 
summed to provide the total risk severity rating, point score, indicated in the lower right 
corner of the table. This determines where the point total falls on the scale shown and 
identifies the programs overall risk rating of using commercial items. In turn, each 
individual category risk rating could be determined by calculating the responses for that 


individual category, based on the number of questions for that category. 


If most of your responses were a's, your organization has a low risk profile for 
successfully implementing a commercial application package(s). While an overall low 
risk is a strong indicator, it is important to note that this does not mean there is "no-risk". 


Every commercial product(s) implementation involves some degree of risk. 


If most of your responses were b's, your organization has a moderate risk for 
implementing a commercial product(s). Carefully examine the questions, particularly 
with medium risks (b) and high risks (c) responses to identify specific vulnerabilities and 
record them in a database or risk mitigation plan with action items or task plans to track 


risks to closure. 


If most of your responses were c's, your organization has a high degree of risk for 
implementing a commercial product(s). Review the questions to help your organization 
identify critical areas that need to be reexamined regardless of its commercial 


implementation phase. 
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Total Calculations a’s b’s 


Totals from Process: — 
Totals from Technology: — 
Totals from Implementation/ 


Support oo 


Program Totals 
Totala’s _—_— x (1) 
+Totalb’s x (2) 
+Totale’s _—x (3) 
Project Total= > 


Table 3. Commercial Item Risk Profile 


D. SUMMARY 


This chapter described a method for facilitating the systematic and repeatable 
identification of risks associated with use of commercial items. It presents a 
questionnaire, checklist, to start the managers and engineers thinking about and planning 
on how to avoid, mitigate and accept the risks inherent in any software project using 
commercial items. Risks that are not identified have the greatest chance of becoming 
problems. Many organizations who attempt to implement a commercial products(s) 
without sufficient analysis and preparation encounter significant challenges that can be 
related to the business processes used to build systems, technologies used to construct the 
system, and logistical support issues that inevitably arise. As a minimum, the project 
manager, software manager, system engineer/manager, any software technical leads, and 
the software engineers, should fill out and discuss the risk identification method stated in 


this chapter. Careful consideration of these issues will help to minimize your 
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organization's risk severity rating and curb future expenditures. With any level of risk, 
awareness of lessons learned by other organizations that have implemented commercial 
items will help build or strengthen strategies to address any unexpected challenges that 


may arise. 
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V. COTS RISK QUESTIONNAIRE ANALYSIS 


A. INTRODUCTION 


The questionnaire was sent electronically to the Office of the Secretary of 
Defense, Command Control and Communication (C3I) Commercial Policies and 
Oversight Office, the Army’s Communication and Electronics Command (CECOM), the 
Marines Systems Command along with numerous other individual project offices that 
utilize commercial products to elicit responses, capture experiences, and record the 
results of the questionnaire for active Department of Defense (DOD) projects that utilize 
commercial products. The intent was for the major commands or organizations to 
distribute the questionnaire to project offices within their organizations that use 
commercial products to complete, revealing the highest risks for their projects. The 


following are responses from active programs using commercial product(s). 


B. DEFENSE LOGISTIC AGENCY (DLA): BUSINESS SYSTEMS 
MODERNIZATION 


The Defense Logistic Agency (DLA) is the primary logistics provider for the 
DOD and is continually seeking ways to improve and reduce the cost of distribution. It is 
undergoing a major Information Technology and reengineering transformation, Business 
Systems Modernization (BSM), to modernize the agency’s business practices by using 
best DLA and commercial practices and commercial software; thus, allowing them to 
rely on industry for support and reduce inventory levels by hundreds of millions of 


dollars. 


The new information technology system being implemented allows DLA to 
exploit new emerging technologies and streamline its supply chain process by 
consolidating its operations to one level of national inventory, generating great 
economies of scale as well as total visibility of all DOD stocks. Letting them achieve the 
proven benefits of commercial software, provide better service at higher workloads, 
reduce the cost, and pass the savings of this improved process back to their military 


customers. 
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The questionnaire was completed by an individual with over 10 years of 
experience on building systems using commercial products; however, his experience was 
with minor projects, not one of this magnitude (enterprise-wide). Some of the lessons 


learned from acquiring and developing commercial product(s) for this project are: 


° Do not modify core COTS software 
° Willingness to adapt 
° Completeness of requirements 


Table 4 provides the risk profile for DLA’s BSM project with the following 
factors being identified as a high risk. The complete results from the questionnaire are 


contained in Appendix C. 


° Many functions supported by the commercial product 

° Very complex interfaces between commercial product and other systems 

e Many of the interfaces must remain unchanged 

e Extensive training required to operate and maintain the commercial 

product 
Total Calculations a’s b’s c’s 
Totals from Process: — an aaa 
Totals from Technology: 1 —— —— 
Totals from Implementation/ —— —— —_1_ 
Support 
Totals | 14 | Js | 


Program Totals 
Total a’s 22 x (1) 
+ Total b’s 14 x (2) 
+ Total c’s 5 x (3) 
Project Total = 65 


Table 4. DLA BSM Commercial Item Risk Profile 
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C, ARMY HUMAN RESOURCE SYSTEM (AHRS) 


The Army Human Resource System Product Management Office provides and 


maintains the personal management information system for the active Army, the Army 


Human Resource System (AHRS) Super Server. It is an integrated automated field 


military personnel management system designed to: 


Serve America's Army during mobilization, war, and demobilization 
Serve the Active Army during peacetime 


Provide commanders a responsive personnel management system, which 
facilitates peacetime personnel strength accounting management and 
wartime operations. 


The questionnaire was completed by an individual with over 30 years of 


experience on building at least 20 systems using commercial products. Some of the 


lessons learned from acquiring and developing commercial product(s) for this project are: 


Do not use software that does not have a long standing commercial user 
base 


Do not allow GOTS products to be forced onto your program. These are 
generally built with commercial products no longer in business 


Table 5 provides the risk profile for the Army’s Human Resource System project 


with the following high risks factors being identified. The complete results from the 


questionnaire are contained in Appendix D. 


Many functions supported by the commercial product 


Much customization of the commercial product needed to meet the needs 
of the organization 


Testing approach was designed for traditional testing of a system 
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Total Calculations a’s b’s c’s 


Totals from Process: 17 1 0 

Totals from Technology: —— —— = 

Totals from Implementation/ a= ese 2 
Support 6 4 1 


Totals 


Program Totals 
Total a’s 30 x (1) 
+ Total b’s 7 x (2) 
+ Total c’s 3 x (3) 
Project Total = 53 


Table 5. Army’s Human Resource System (AHRS) Commercial Item Risk Profile 


D. ARMY (CECOM) COMMUNICATIONS SOFTWARE ENGINEERING 
SUPPORT DIVISION (CSES) 


The Communications Software Engineering Support (CSES) Division provides 
life cycle software engineering services to the Program Executive Office for Command, 
Control, and Communications Systems (PEO C3S), as well as other Department of the 
Army and DOD organizations and agencies. These services include all activities 
necessary to ensure the reliability, maintainability, interoperability, and configuration 
integrity of the software components used in communications and related Mission 
Critical Defense Systems (MCDSs) for both systems under development and systems 
deployed to operational units worldwide assuring joint interoperability of tactical 


switching and network management software/firmware through quality assurance testing; 
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oversight and certification recommendation of all new software releases; supporting 


acquisition of interoperable tactical switching and network management systems; and 


serving as the single point of contact for the warfighters in all matters involving switch 


interoperability. 


The questionnaire was completed by an individual with four years of experience 


making recommendations on commercial product(s) that were no longer supported or 


reached their end of life with two recommendations for commercial replacements being 


integrated. Some of the lessons learned from acquiring and developing commercial 


product(s) for this project are: 


Known your requirements well 


Assess and evaluate different available commercial products based on the 
requirements well in advance 


Close, continuous, and active partnership among the vendor, customers, 
developer, and most importantly the users 


Table 6 provides the risk profile for the Army’s Human Resource System project 


with the following high risks factors being identified. The complete results from the 


questionnaire are contained in Appendix E. 


Vendor unknown or poor performance in integrating the commercial 
application 


Vendor has a track record of exceeding total life cycle cost estimates 

No discussion with the vendor for future plans of the commercial product 
System is a new system 

Very complex interfaces between commercial product and other systems 
No flexibility in the design to allow future changes in functionally 


Program has not gather information from other organizations that 
implement commercial applications 


No performance measures to measure the effectiveness of the commercial 
product 


No contingency plan for commercial vendor going out of business 


Extensive training required to operate and maintain the commercial 
product 
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Total Calculations a’s b’s c’s 


Totals from Process: 
Totals from Technology: 
Totals from Implementation/ 


12 
i 
Support = 


3 2 
= =a 
ona — 


Totals 


Program Totals 
Total a’s 5 x (1) 
+ Total b’s 26 x (2) 
+ Total c’s 10 x (3) 
Project Total = 87 


Table 6. Army’s Communications Software Engineering Support Division 
Commercial Item Risk Profile 


E. ARMY GLOBAL COMBAT SUPPORT SYSTEM (GCSS) 


Global Combat Support System (GCSS)-ARMY is the largest and most complex 
information technology program in the Army, which will, over time, replace or interface 
to all of our existing CSS automated systems and provide automatic, user-transparent 
communications for routine transactions. It will encompass personnel, financial, medical, 
and other non-logistics CSS functions and be made up of a series of functional modules 
(or Product Lines) such as Supply, Property, Maintenance and Management with each 
module operating within the Defense Information Infrastructure and run at any level or 


organization where the Army performs that function. 


Designing the modules at the same time gives the modules a common look and 
feel using a graphical user interface with point and click techniques with Commercial- 
Off-The Shelf hardware and the Windows NT operating systems used to support the new 
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software. These two features coupled with embedded training will simplify initial and 


sustainment training requirements tremendously. 


The questionnaire was completed by an individual with one year of experience on 
building systems using commercial products with no lessons learned and since the 
program is under implementation with over 130,000 expected users with multiple and 
different training requirement; there, he could not answer questions like “how efficient is 
it?” 

Table 7 provides the risk profile for the Army’s Global Command Support 
System project. Even though the project has a low profile for successful implementation 
with commercial products, it is important to note that this profile does not mean a "no- 
risk" profile. Every commercial product(s) implementation involves some degree of risk. 


The complete results from the questionnaire are contained in Appendix F. 


Total Calculations a’s b’s es 
Totals from Process: —L —0_ a 
Totals from Technology: —l —o_ at 


Totals from Implementation/ 
Support ee ee 
Totals 


Program Totals 
Total a’s 39 x (1) 
+ Total b’s 1 x (2) 
+ Total c’s 0 x (3) 
Project Total = 41 


Table 7. Army’s Global Combat Support System Commercial Item Risk Profile 
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F. MARINE CORPS COMBAT VEHICLE TRAINING SIMULATOR 


The Marines want to be able to use the simulators to cover the full breadth of 
training from the beginning driver all the way to the tactical driver, who is in severe off- 
road conditions and inclement weather and blackout conditions. Too often, driving 
simulations are based on flight simulations or video games, and therefore are 
unsatisfactory for serious training. The simulator will not only display exterior driving 
conditions, but also will provide a realistic environment of the interior of the vehicle. 
Everything the driver will have available to him in the vehicle will be available to him in 


the simulator. 


A Semi-Automated Forces (SAF) demonstration system will be developed that 
will be networked with Raydon-developed LAV-25 and M1A1 Abrams Tank simulators 
via the High Level Architecture (HLA) protocol be and operate in two basic modes. As 
an exercise editor, it is used to define entities and their characteristics, build a scenario 
containing a selection of those entities and assign appropriate behavior to those entities. 
As aruntime engine and Situation Awareness Display, the SAF controls the behavior of 
its entities and displays the composite worldview of all entities in the simulation, 
including those external to the SAF. There will also be three visual databases: a desert 
database, a geotypical European database and a geotypical rural database to support 


training in both Visual and Thermal modes. 


The questionnaire was completed by an individual with five years of experience 
building systems with commercial products; however, she has never participated in 
selecting commercial software for the integration into a system. Table 8 provides the risk 
profile for the Marine Corps’ Ground Transportation Engineer Systems project with the 
following high risks factors being identified. The complete results from the questionnaire 


are contained in Appendix G. 


e No data right negotiated into the contract 
° Uncertain about what licensing agreements are needed 
° Much customization of the commercial product needed to meet the needs 


of the organization 


° Many functions supported by the commercial product 
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e Program has not gather information from other organizations that 
implement commercial applications 


° No performance measures to measure the impact and effectiveness of the 
commercial product 


e Extensive training required to operate and maintain the commercial 
product 
e Very little training resources available to the customer 
e Testing approach was designed for traditional testing of a system 
Total Calculations a’s b’s c’s 
Totals from Process: 5 11 2 
Totals from Technology: 
, 3 5 2 
Totals from Implementation/ —— = — 
Support 2 =e > 


Totals 


Program Totals 
Total a’s 12 x (1) 
+ Total b’s 20 x (2) 
+ Total c’s 9 x (3) 
Project Total = 79 


Table 8. | Marine Corps Combat Vehicle Training Vehicle Simulator Commercial Item 
Risk Profile 


G. ARMY COMMON SOFTWARE PROGRAM 


The Army's Common Software Program is based upon the Global Command and 
Control System (GCCS) which has two main objectives: the replacement of the World- 


Wide Command and Control System (WWMCCS) and the implementation of the 
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Command, Control, Communications, Computers, and Intelligence (C41). For the 
Warrior. GCCS is designed to become the single, global command, control, 
communications, and intelligence system to support the war fighter, whether from a 
foxhole or from a command center. The GCCS system is based upon the Common 
Operating Environment (COE) which provides the infrastructure for all command and 
control systems. This COE consists of an integrated architecture made up of hardware 
and software that provides standard, modular, system support and applications support 
software for a set of functional applications. The COE software is a multi-layered open 
system architecture consisting of modular functional applications, application support 
software, standard system support software which is designed to operate on a standard 
suite of computers and consists of 19 functional areas. It fully supports a reuse program 
that is domain specific, architecture centric, and systematic, implementing the 


Department of Defense (DOD) software reuse vision and strategy. 


The questionnaire was completed by an individual with two years of experience 
building systems with commercial products and has participated only once in the 
selection of software components that were later adapted or integrated. Some of the 
lessons learned from acquiring and developing commercial product(s) for this project are: 


e Never rely on a single vendor for critical functionality, always have 
alternate products lined up 


e Consider the likelihood that the vendor will not be there to support it in the 
future 


Table 9 provides the risk profile for the Army’s Common Software program with 
the following high risks factors being identified. The complete results from the 


questionnaire are contained in Appendix H. 


e The implementation team has no experience with the commercial product 
° Many functions are supported by the commercial product 

° New or immature commercial product 

e Program has not gather any information from other organizations that have 


implemented commercial products 
e Testing approach was designed for traditional testing of a system 


° No contingency plan for commercial vendor going out of business 
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e Other contractors supporting the organization in functions affected by the 
commercial product have no experience (new commercial product) 


Total Calculations a’s b’s 


Totals from Process: 9 

Totals from Technology: 

Totals from Implementation/ 
Support 




















otals 


Program Totals 
Total a’s 19 x (1) 
+ Total b’s 15 x (2) 
+ Total c’s 7 x (3) 
Project Total = _70 


123 82 41 
Table 9. ARMY Common Software Commercial Risk Profile 


H. SUMMARY 


The questionnaire was implemented on current DOD projects that utilize 
commercial products. It provided to be an effective and efficient tool that could be 
applied by the program manager or decision maker in a consistent and systematic manner 
to assist them with the early identification and prioritization of risks associated with 
commercial products. This provides the program manager or decision maker with 
sufficient information and expands the time they will have to mitigate and resolve the 
risks by letting them make wise decisions and apply proper resources, based on the 


prioritization, early in the process and use their creativeness to properly mitigate and 


resolve each risk. 
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Identifying and understanding the risks of commercial products is the first step to 
ensure that the acquiring activity can minimize downstream surprises and problems and 
achieve the benefits of using commercial products. Shrinking budgets and tighter 
schedules virtually eliminate any margins that could be retained to offset problems that 
might occur later in a program. The next chapter provides some strategies and offers 
suggestions to help organizations manage and mitigate the risks associated with 


commercial products. 
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VI. MITIGATION STRATEGIES AND SUGGESTIONS 


A. INTRODUCTION 


Once the risks have been identified, mitigation strategies to reduce or eliminate 
the risks need to be developed and managed. Even though there are many risks, there are 
specific ways that the risks can be reduced. This section describes some mitigation 
strategies and offers some techniques/suggestions to help organizations tackle the 
risk/challenges listed in the previous chapter. 

B. SYSTEM REQUIREMENTS 

1. Flexible and Negotiable 


A traditional development model that specifies all system requirements prior to 
considering the capabilities available in the marketplace is ill suited to the development 
of systems incorporating commercial items. Since product development is based on 
commercial market needs and is under the manufacturer’s control, requirements must be 
written with a quantifiable range of acceptable performance limits (e.g., high and low 
values) offering flexibility and letting the integrator make the best possible match (within 


constraints) between commercial product capabilities and the requirements. 


The requirements should also be prioritized with criteria established to distinguish 
absolute requirements (must have) from the less absolute (nice to have) requirements; 
providing the needed flexibility when selecting among a variety of commercial product 
candidates. This flexibility is especially important with commercial products since they 
are sold and supported by manufacturers in an “as is” state which may not meet all the 
requirements as stated. Government and commercial programs that were successful in 
incorporating commercial products were able to trade-off requirements with the 
operational and acquisition communities in order to achieve a best value solution. For 
example, The AWACS program eliminated salt spray requirement to facilitate the use of 
commercial computers, while the NSSN submarine structure was modified to provide 
environmental isolation [USAF00]. Adapt the requirements to meet the commercial 
product is better than modifying the commercial product to the requirements to meet the 


requirements.. If a commercial product is modified to meet a requirement, the warranty 
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would be voided and a unique product may be created that requires uniquely developed 
(and often more costly) life cycle support. 


2. Examine Requirements Gap 


Ideally, the commercial product(s) functionality should closely match the 
intended use. However, because no commercial product(s) have been specifically 
designed to meet your organization's unique requirements, there will be a gap between 
the business processes supported by your existing systems and those supported by the 
commercial product(s). It is imperative that you understand this gap well before the 
implementation begins. If this gap is too great then more effort will be expended 
developing adequate interfaces than developing the product from scratch. 


3. Involve Users 


Because the implementation of a commercial product significantly impacts the 
functions of an organization, it is imperative to involve the user community in the 
planning process from the outset. Early end-user involvement is a common risk 
mitigation strategy to ensure that the requirements accurately reflect user needs. User 
familiarization allows for requirement prioritization and the early identification and 
resolution (trade-offs) to minimize user acceptance issues and avoid costly changes and 
delays during system development and deployment activities. It also allows the user 
community the time to become familiar with commercial technologies that are available 
for meeting their needs. Once a system is placed in-service, formal user participation is 
continued as the system evolves and undergoes changes and updates requiring 
commercial products. In addition to the technical issues, understanding the business 
issues will lower the risks associated with the commercial implementation. A stable 
operating environment coupled with the users willing to accept a new way of doing 


business will also minimize implementation obstacles. 
Suggestions 


The following suggestions can be helpful for the requirements specifications of 


commercial items: 
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To develop flexible and negotiable requirements: 


Requirements must be written with a quantifiable range of acceptable 
performance limits. 


Identified, prioritized, and negotiated all absolute requirements (must 
have) and plan to make repeated tradeoffs among the requirements and the 
capabilities in the marketplace. 


Document all tradeoffs made. 


To bridge requirements gap: 


Determine the gap between the capabilities and services provided in the 
marketplace and those required by the system. 


Include the vendor in tradeoff discussions when possible. 


Provide incentives to encourage the contractor to investigate all solutions 
that lead to the appropriate outcome. 


Ask vendor to conduct demonstrations. 


Don’t modify the commercial item. Encourage vendors to change product 
capabilities and performance to meet your requirements only if the change 
will be incorporated into the commercial product(s). 


To Involve Users: 


Communicate and cooperate early with the user to ensure flexibility in the 
system requirements and to share knowledge of potential commercial 
product(s) that may be available to meet requirements. 


Provide early functional demonstrations, prototyping to provide user 
hands on familiarization with the capabilities of the candidate commercial 
product(s) and get user buy-in on contract requirements. 


EVALUATION OF COMMERCIAL PRODUCT (S) 


It is critically important to evaluate all aspects of a commercial item. In some 


cases, commercial item evaluations are performed as part of source selection. This is a 
highly constrained form of evaluation that must be conducted only in accordance with 
source selection criteria and the source selection plan. However, the definition of 
evaluation applied in this document is far broader. Evaluation is also necessary to assist 
in identifying commercial capabilities such as security and information assurance, inter- 
operability, reliability, and maintainability when choosing among alternate architectures 


and designs, in determining whether new releases continue to meet requirements, and in 
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ensuring that the commercial items function as expected when linked to other system 
components. These forms of commercial-item evaluation provide critical information 
about the tradeoffs among system context, architecture and design, and commercial 
capabilities. Unfortunately, evaluating commercial items in order to identify system 
tradeoffs is an unfamiliar process for many program managers (and their users). It is 
equally unfamiliar for many contractors who are more comfortable with simply meeting a 
specified set of requirements. 


1. Market Research 


Successful systems that incorporate commercial items recognize that, as 
customers, they must be informed and assertive to maximize the benefits of using 
commercial items [ALBE03] by attempting to gain leverage on the vendors through 
market research. Market research is a process of collecting information about existing 
and emerging technologies, products, manufacturers, suppliers and their trends. It 
consists of market surveillance and market investigation. Market surveillance is a 
continuous canvassing of the commercial market to identify existing and future 
technologies, vendors’ products and market trends. It can include Internet searches, 
attending trade shows, reading technology publications, hiring consultants, visiting 
manufacturer/supplier facilities and product demonstrations. Market investigation is a 
more focused process of identifying and determining if specific commercial items can 
meet particular functional requirements. It involves not only identifying potential 
alternatives for investigation, but also identifying any deficiencies in the commercial item 
that would require modification, and then determining the extent of that modification. 
Extensive modification of commercial items within a program would take the product out 


of the category of a commercial item, thus increasing the program’s risk. 


Market Investigations also includes proactively planning for the continued 
support or replacement of soon-to-be obsolete products, identifying the product(s) end of 
life (EOL) and end of service (EOS) dates. One program selected commercial items with 
the expectation that the vendor would provide the necessary maintenance capabilities. 
However, the vendor’s commercial support strategy did not provide the spares, training, 
or repair cycles necessary for military use. The program was left with a choice: redesign 


the system or buy the additional capability. [USAF00] 
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2. License and Data Rights 


Licensing is the primary vehicle for securing the use of commercial items such as 
software; data rights are marketplace vehicles for protecting a vendor’s intellectual 
property. Understand completely, the details associated with the product contract, 
including the licensing agreement. Vendors offer different licensing agreements and you 
need to select the agreement appropriate to your program and circumstances. For 
example, if everyone in the organization will need to access the product, ensure the 
license is for the entire enterprise. Be sure to find out: who owns the license to the source 
code; what rights are provided relative to source code modification; and what 
arrangements will exist at contract expiration. One program expressed frustration that the 
de facto selection of a commercial item had already been made prior to release of the 
solicitation because of the beneficial pricing arrangements from previously negotiated 
enterprise licenses. While the larger organization saved money in negotiating one set of 
licenses covering use by many programs, this practice limited the individual program’s 
flexibility in choosing the most appropriate commercial item for the system. Another 
program neglected to negotiate for all necessary licenses as part of the initial 
procurement. After the commercial item was selected and system development began, the 


vendor’s price for additional licenses increased dramatically [USAFO00]. 


Some commercial products are so critical to the system that the program must be 
protected from a vendor’s potential unwillingness or inability to support older releases of 
the product. Some programs found that an agreement to put technical data in an escrow 
account (rather than purchasing technical data directly) was a cost-effective compromise. 
However, one program never checked that the escrow account was set up and maintained 
by the vendor. When the vendor went out of business, the program was forced to gather 
what technical data it could from personnel who had previously worked for the vendor 
[DODAO00]. On the other hand, successful programs negotiated terms of the escrow to 
include the essential data and contingencies, audited the escrow account regularly to 
make sure the data was current and complete, and budgeted for the cost of the escrow 


throughout the life of the system. 
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3. Vendor Relationship 


It is incumbent on the program manager to determine how important the program 
is to a specific vendor as part of the commercial-product evaluation. The relationship 
between the program manager and the vendor is, in most instances, very different from 
the relationship with a contractor. While contract incentives shape the relationship with a 
contractor, the vendor is selling a product. Program managers should obtain commercial 
product(s) from vendors who view the software as one of their products. If a vendor does 
not consider an item to be part of its business, and/or developing software products is not 
the main business of the vendor, then its development, release management, and 


documentation practices may not be adequate. 


Program managers need to develop a trusting, but contractual and mutually 
beneficial relationship with the vendor. Finding vendors with the best quotes and support 
services are time consuming and not an easy process. Assess the vendor’s past 
experience employing commercial products. When both government and the vendor 
were experienced in applying commercial products, the success rate was high and cost 
savings were dramatic. When both were inexperienced the success rate was very low and 


costly, many times resulting in substantial overruns and even program failure [FAAC02]. 


Consider the characteristics of the vendor as part of the process. Examples of 
characteristics to look for include company size, their level of establishment, how long 
they have been selling the product, its level of support (does it continue to support 
customers using significantly older versions of their product as well as the customers 
using the latest version), and is the company willing to work with the organization rather 
than for it (will they maintain their autonomy and listen to the organization's requests for 


enhancements rather than automatically accepting that such changes must be made’?). 


Figure out the leverage you have and how you can influence the vendor to be 
responsive to unique program needs and incorporate new features into the commercial 
item. Some program managers have expressed frustration that vendors do not react to 
program needs and direction. Other program managers have tried to use the same 
techniques with vendors that had been successful when applied to contractors and 


subcontractors—usually with disappointing results [DODA00]. The degree of leverage 
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you can apply to your vendors varies depending on the size of your program and the 
presence your organization has in the industry. The larger your organization the more 


pressure you can exert in getting vendors to meet your projects requirements. 


One approach to this is to join the vendor users' group and the appropriate 
professional and standards bodies. Joining the users’ group is valuable as it provides the 
organization some insight into the use of the product in other parts of the marketplace. 
Future directions of the product may be discussed in these meetings, providing the 
organization with the opportunity to express their needs in the domain the product 
addresses. Joining the relevant professional and standards bodies is important so that the 
program can remain current with the direction being taken that affects their system. 
Further, by being active in the professional and standards bodies, the organization is able 
to influence (although not control) the future course of the business approach so that the 
business practices embodied by the philosophy are more likely to remain consistent with 
the organization's needs. 


4. Testing, Evaluation, and Validating Reliability 


Given the lack of technical information about commercial product(s) (i.e., “black 
box” — undisclosed designs) and the variety of product types that change rapidly, testing 
potential commercial product(s) is a necessary step in product evaluation to ensure that 
operational suitability; reliability, availability and maintainability requirements are met, 
since manufacturer claims about the capabilities tend to be optimistic. More effective up- 
front planning of independent test and evaluation is needed to ensure that enough data is 
obtained to fully evaluate the capabilities. Exercise a healthy skepticism of commercial 
product claims by testing candidate products using an application testbed to verify 


features and to support your selection decision. 


Full system-level testing must be performed in a test facility that provides or 
emulates the external interfaces and actual operating environment in order to verify 
operational suitability, effectiveness and performance and should never be waived. This 
raises the probability that the commercial product(s) perform as they did in the 
development environment and that they do not introduce any unknown performance 


characteristics into the interfacing systems. As engineering changes are introduced into 
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the continuously evolving system, a dedicated testing environment must be maintained to 
replicate the integration steps and determine if the new product or integrated change has 


affected the performances of the system. 


Confirm, with other users, the product's capabilities, especially performance, 
reliability, and scalability by ensuring that the product's capabilities support the needs of 
your organization. For instance, confirm that the product has previously supported the 


number of users and geographic locations your organization will require. 


Reliability requirements must be established early in order to insure adequate 
testing and verification of the reliability of commercial items. Since a commercial item 
has already been designed and developed, and its reliability already established by the 
vendor, the reliability verification should be an assessment of the product within the 
military wartime environment in which it will be used. Testing the commercial item in 
your operating environment to verify that the item’s reliability is meeting the user’s 
requirements. Lower reliability greatly impacts the support costs, system availability, 


and thus the mission accomplishment. 
Suggestions 


The following suggestions can be helpful for evaluating commercial items: 
° Employ outside experts to support program-office evaluation activities. 


° Train the program office and the stakeholders/users on how to evaluate 
commercial items. 


° If possible, obtain hands-on experience with the system. Consider 
prototyping or piloting the commercial item in your environment. Ask 
vendor for a demonstration. 


e At a minimum, visit another organization that is operating the same 
software. 
e Decide in advance what information you want to gain from the evaluation 


of a commercial item. 


e Unless it is impractical, evaluate potential commercial items in a system 
test bed. Do not consider buying any commercial product you haven't 
demonstrated in house. 
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To understand the marketplace: 


Conduct market research independent of the contractor to capture current 
information and base market research decisions on fairness, competition, 
and ethics. 


Participate in the relevant conferences, trade shows, and _ user, 
professional, and standards groups. 


Allocate resources for marketplace activities. 


Proactively anticipate obsolescence situations due to rapid and 
asynchronous product changes. 


Assess the total system operation and support effectiveness, not just 
system performance by determining the projected manufacturer support 
period and inventories for a particular product. 


Select a reputable and reliable vendor that plans to be available to support 
the product. 


To help with Licenses and Data Rights: 


Thoroughly understand commercial products licensing terms and 
conditions to ensure warranties are enforceable. 


Contracting officers should consider including contract options(s) for parts 
and technical buyouts to support future logistics requirements. 


Licenses that transfer to the government/maintainers. 
Volume discounts. 


Put technical data into escrow accounts. Regularly audited the accounts 
to make sure the data I complete and current and complete. 


To strengthen vendor relationships: 


Verify the claims made for commercial items by vendors and contractors. 
Verify the availability of commercial items. 


Examine any acquisition strategy to see where it can be made more 
flexible or better suited to the unique commercial aspects of the system in 
question. 


Use contract incentives to encourage appropriate relationships . 


Maintain close relationships with vendors to exploit improvements and 
avoid surprises. 
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To test commercial items and validate reliability: 


Do not rely on vendor claims, verify with operational demonstrations as 
early as possible by evaluating the potential commercial products in a 
system test bed to ensure product(s) are compliance with performance 
requirements. 


Focus test beds on high-risk items. 


Test for unanticipated side effects in areas such as security, safety, 
reliability and performance from commercial-item upgrades. Ensuring all 
system configurations (more are possible with commercial product use) 
can be recreated; and determine if new manufacturer changes to a 
commercial product configuration cause any unforeseen impacts (ie., 
“unknown-unknowns” to system performance). 


Ensure that performance pass or fail criteria are clearly specified in the 
contract. 


Contracting offices should require contractors to fully disclose item 
reliability data. 


Test Organizations should validate the reported reliability of commercial 
item components and test them thoroughly in new _ operational 
environments. 


Select a mature product. Implementing a commercial item with a 
successful track record is less risky than one that involves new, unproven 
capabilities. 


D. TECHNOLOGY 


The technology as well as your ability to deal with it may both be immature and 


change rapidly. Planning ahead for technology insertion should be an integral part of 


your program. One has to predict the change cycle for each imbedded commercial 


product and plan for regular refreshment of the system throughout the design, 


development, production and sustainment phases of the program. 


1. 


Integration 


Integrating commercial product(s) into a functional system presents new 


challenges and projects cannot be treated as normal low-risk commercial item 


acquisitions. Different vendors write different components with different needs in mind 


that may need to be adapted to work properly. Integration efforts may not only require 
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research and development, but also may need both a demonstration and a full-scale 


development phase before production 


Although the expertise is growing, relatively few programs or contractors have 
extensive experience integrating commercial items into DOD systems. Knowledge of 
both the system context and each selected commercial item is necessary. One program 
assumed that heterogeneous commercial items could be integrated with relatively 
minimal effort. The program neglected the hard engineering work needed to develop 
realistic integration and test schedules, to specify acceptance criteria for the system, or to 
plan for long-term system evolution [DODA00]. These oversights resulted in unhappy 
users, finger-pointing between the vendors and the program office, and cost and schedule 
overruns. Several other programs found that unique technical expertise was required to 
integrate commercial items because the internal architectural and usage assumptions of 


the items were unknown. 


Three integration techniques for commercial items are: 


° To wrap the items in a software container. 

° Use glueware to mediate item interactions. 

e Using bridges or adaptors to smooth over incompatibilities in the item 
interfaces. 


All of these are “black-box” techniques that can be applied without access to the 
commercial product(s) source code. Wrappers are software containers used to mediate 
access to the commercial items. They can be used to force compliance to programming 
standards, provide a standard interface to the commercial product(s), restrict the available 
functionality of the commercial item, or can be upgraded or swapped out with a different 
vendors commercial item [MAUROO]. Glueware is used as middleware to bind 
components together. It can be used for control flow, to invoke the item’s functionality 
and do exception handling. This can also include code to resolve incompatibilities 
among commercial items [MAURO00]. By acting as an adaptor or bridge, the glueware 


can allow the interaction of two items [MAUROO]. 
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2, Obsolescence and Upgrades 


Many organizations or program office assume they can avoid the problems 
associated with upgrade by simply continuing to deploy an older release of a commercial 
item. While this may be true for some hardware items, it is rarely the case for software 
items, where new and desirable capabilities and performance are frequently added, bugs 
are fixed, and vendors drop maintenance for older releases. A release or two can 
sometimes be safely skipped, but most software commercial items (and many hardware 
commercial items) must eventually be upgraded. Except in very specific cases, DOD is 
normally ill prepared to implement the necessary changes to old versions of commercial 
items in order to avoid technical obsolescence and keep them functioning, even when 
good technical data is available. Several programs were successful by deliberately pre- 
planning for frequent upgrades of commercial items, technology insertion, and retirement 
of obsolete items [DODAO00]. Of course, even the most careful planning cannot 
anticipate all exigencies, such as a vendor going out of business or being taken over by a 
larger firm with different priorities 


3. New and Strong Configuration Management Techniques 


The rapid evolution and proprietary nature of commercial products/systems 
require a robust and diligent configuration management (CM) program. Frequent 
changes to commercial items have caused many programs to maintain multiple 
configuration baselines both during development and in the field. This places unusual 
demands on traditional configuration-management processes that strive to maintain a 
single configuration baseline. Several programs that depended on multiple commercial 
items found that some items required specific versions of other items in order to interface 
effectively. Upgrading one commercial item caused a chain reaction that demanded 


changes to other commercial items within the system [DODAO00]. 


Unlike custom developed systems, the government has no control over the speed 
and content of product configuration changes since the commercial product 
manufacturers control them. Vendors release items according to their own schedules, and 
many programs found that individual sites were not always willing to upgrade to the 


latest version. This requires programs to possess a configuration-management system 
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that lets then select from among multiple versions of commercial items in order to 


construct different system configurations. 


Commercial products are proprietary to the manufacturer and get documented at a 
higher level (source control drawings, specification sheets, inputs/outputs, etc.) resulting 
in limited information on manufacturing processes, internal design, components, etc. 
Included with this higher level of documentation are different numbering conventions by 
the manufacturers. These differences shift CM focus from controlling configurations (as 
with custom development programs) to managing commercial product and system 


configurations (at the manufacturer-controlled product level). 
Suggestions 


The following suggestions can help minimize the impact of commercial 


technology: 


To integrate product(s) 


e Establishing an ongoing market research effort that includes market 
surveillance (technologies, trends, available manufacturers and products, 
etc.) and market investigation (product testing and obsolescence 


projections). 

° Developing/delivering integrated technology evolution planning data, 
conducting working group meetings and providing status at program 
reviews. 

e Base interfaces on publicly recognized industrial standards that are widely 


supported in the marketplace. 


e No vested interest in any one particular manufacturer or commercial 
product set. 


To manage change: 


° Ensure that rigorous configuration management is exercised. 

° Monitor the marketplace for technology advancements. 

° Establish plans to work with vendors for problem resolution. 

e Periodically check with vendors for planned software upgrades/updates to 


commercial products. 


° Developing/delivering periodic (every four months) system product 
obsolescence projections, impacts and solutions. 
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e Establishing a test facility capable of continuously testing commercial 
products for compatibility, compliance and conformance. 


E. AVOIDING MODIFICATION 


Modification of commercial products can involve the addition or deletion of code, 
changes to the hardware design, or changes to any of the product support (documentation, 
spares, etc.). Commercial products must be used “unmodified” to retain the value of a 
commercial product; otherwise, voided warranties, lack of support, and upgrade 
difficulties will result. If this strategy is ignored the program runs the risk of incurring 
additional support costs and supportability issues that may be less cost effective the 
custom design. The savings in development costs and schedules would be offset by the 
modifications of the commercial products resulting in a unique version of the product that 
the manufacturer will not support under warranty and must be supported separately from 
other versions, often with increased support costs. One private corporation fell into the 
trap of modifying most of its commercial items in order to give them a unique corporate 
flavor. As a result of the practice, many of the corporate programs modifying 
commercial items experienced recurring technical problems and cost overruns 


[DODAO00]. 
Suggestions 


The following suggestions can help organizations avoid modifications to 


commercial product(s): 


° Persuading the manufacturer to incorporate the acquiring activities unique 
requirements as part of the commercial product’s functionality; 

° Verify that the contractor proposes to use the commercial product without 
modification 

° Requiring that notification of proposed COTS modifications be made only 


with trade-off considered and Government consent; 


G. TOTAL OWNER COST (TOC) 


Both commercial and DOD programs frequently underestimate the unique 
sustainment costs associated with commercial items. These costs include market 
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research, evaluation, test and integration for version upgrade, commercial-item 
replacement, technology refresh, and annual licensing fees. Programs must deal with 
change and expense elements throughout a product lifecycle that may not be evident in a 
custom product. Successful programs have generated strategies for assessing items such 
as product deviations caused by commercial product evolution cycles, extended military 
product support and licensing against the expected benefits of more affordable sparing, 


shorter development times and increased performance [USAF00]. 
Suggestions 


The following suggestions can help organizations identify Total Ownership Cost 
(TOC): 


° Identify and budget sufficient funds and staff for monitoring current and 
emerging commercial product(s) and technology — market research, 
integration lab, testing facility, license renewal and data rights, and 
reacting to new product releases -version upgrades in annual budgets 


° Incentive the prime contractor to provide a credible estimate of support 
costs 

° Use multi-year, unrestricted contracting could potentially reduce costs 
even more. 


H. SUMMARY 


Unfortunately, there are no silver bullets to resolve the risks/challenges associated 
with commercial items. Early identification of the risks associated with commercial 
items and the techniques and suggestions discussed in this chapter provide an effective 
approach that can be incorporated into an overall risk management program for systems 
employing commercial items. Such a risk mitigation approach allows the acquiring 
activity to benefit from commonly experienced government and industry lessons-learned. 
Personnel will become better educated, trained, and effectively employ commercial items 
by actively soliciting and rigorously incorporate into their plans those lessons learned 
from organizations similar to theirs and by exploring products before selecting them, 
talking to some of the product's other customers and understanding the product's 
customer base. Allowing them to effectively and efficiently employ commercial items 


within their programs. 
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Vil. CONCLUSION 


The inclusion of commercial items in the acquisition process is recognized as an 
opportunity to save both time and money. But it is not the Holy Grail. Anyone who 
believes that selection of a commercial product and inserting it into a program will be the 
quick fix believes in fairy tales, and does not really understand the process. Commercial 
products can do all that everyone expects them to do, and may be an excellent solution in 
many cases, but their use should be the result of careful analysis and research. There is a 
tendency to assume that a commercial product can be used as-is, without any serious 
thought given to the difficulty and risk involved in the commercial product. It has been 
assumed that the use of a commercial product alleviates all risk of integration, when, in 
fact, just the opposite may occur: commercial products may be even more difficult to 


integrate. 


Every aspect of acquisition planning, system engineering processes, test planning, 
etc. must be explicitly crafted to account for every challenge the commercial product 
presents. The mentality ought to be how we can do it as opposed to why we cannot. 
Everything about the commercial product must be known and understood by those who 
establish the requirements. Not every new requirement can realistically be addressed 
with a commercial product. The commercial products must be chosen carefully with the 
marketplace clearly understood in order to have a flexible range of “requirements” 


sufficient to allow commercial products to qualify. 


The risks associated with traditional system development do not disappear simply 
because the system makes use of commercial products. Risks associated with 
commercial products are likely to change more rapidly, and new risks may arise more 
often than with customary system risks. In order for systems to be successful and achieve 
the benefits of using commercial items, they need to minimize the uncertainties and 
manage the unique risks associated with the commercial product(s). Identification, the 
first step in the process, involves transforming the uncertainties and issues about the 
project into distinct (tangible) risks that can be described and measured. With any risk, 


awareness of lessons learned by other organization that have implemented systems using 
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commercial products will help build or strengthen strategies to address any unexpected 


future challenges. 


This thesis describes a framework, checklist (questionnaire) and provides some 
strategies and suggestions to help organizations manage and mitigate the risks associated 
with commercial products. The checklist serves as a starting point and enhances the 
project manager’s or decision-maker’s risk management abilities by providing a 
systematic and repeatable method, early in the process, for the identification of risks 
associated with the use of commercial products. It uses a set of known risks that are 
classified into three categories: process, technology, and implementation and support. 
Many organizations which attempt to implement a commercial product(s) without 
sufficient analysis and preparation encounter significant challenges that can be related to 
the business processes used to build the systems, technologies used to construct the 
systems, and logistical support and implementation issues that inevitably arise. Each 
category is then further divided into specific elements or attributes to generate the 
projects risk profile. This profile determines what level of impact (high, medium, or low) 
these factors have on the programs that incorporate the commercial product. Managers 
can then prioritize these risks and apply resources to properly mitigate and resolve the 


identified risks. 


To test the checklist, it was implemented on active DOD programs that 
incorporate commercial products into their systems and proved to be an effective and 
efficient tool. Project managers or decision-maker’s were provided with sufficient 
information to identify and prioritize risks early in the process. They could then use their 
creativity along with some of the suggestions to make wise decisions and positively 
impact the success of their programs. [FALV02, FLAN02, HAKE03, PORT03, TRIE02, 
WILLO3] 


While considerable work still remains to be done in developing additional 
identification methods, analysis, planning, tracking, and controlling risks associated with 
commercial products. The project manager or any decision maker, should, as a 
minimum, fill out and discuss the checklist, with the checklist becoming a natural part of 


the project activities. This will force program managers to focus their efforts on the 
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highest impact areas and eliminate fires before they happen, thus, minimizing their 
organization's risk severity rating while curbing future expenditures. Although the scope 
of the thesis ends at this point, it is recommended that the program or project manager’s 
develop a risk plan for each prioritized risks. These identified risks would then be 
monitored and tracked continuously throughout the life cycle of each project. Only by 
understanding and addressing these unique factors imposed by commercial products will 
the program managers be able to attain their benefits. Enhancing their ability to manage 
the risks associated with commercial products and making them more successful in all 
the software development projects they lead. Moving towards market-oriented business 
practices that are better suited to the acquisition and life cycle support of commercial 


(COTS)-based systems. 
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APPENDIX A FAR DEFINITION OF COMMERCIAL ITEM 


(a) Any item, other than real property, that is of a type customarily used for 


nongovernmental purposes and that — 
(1) Has been sold, leased, or licensed to the general public; or, 
(2) Has been offered for sale, lease, or license to the general public; 


(b) Any item that evolved from an item described in paragraph (a) of this 
definition through advances in technology or performance and that is not yet available in 
the commercial marketplace, but will be available in the commercial marketplace in time 


to satisfy the delivery requirements under a Government solicitation; 


(c) Any item that would satisfy a criterion expressed in paragraphs (a) or (b) 


of this definition, but for 


(1) Modifications of a type customarily available in the commercial 


marketplace; or 


(2) Minor modifications of a type not customarily available in the 
commercial marketplace made to meet Federal Government requirements. Minor 
modifications means modifications that do not significantly alter the nongovernmental 
function or essential physical characteristics of an item or component, or change the 
purpose of a process. Factors to be considered in determining whether a modification is 
minor include the value and size of the modification and the comparative value and size 
of the final product. Dollar values and percentages may be used as guideposts, but are 


not conclusive evidence that a modification is minor; 


(d) Any combination of items meeting the requirements of paragraphs (a), (b), 
(c), or (e) of this definition that are of a type customarily combined and sold in 


combination to the general public; 


(e) Installation services, maintenance services, repair services, training 
services, and other services if such services are procured for support of an item referred 


to in paragraphs (a), (b), (c), or (d) of this definition, and if the source of such services — 


fe 


(1) Offers such services to the general public and the Federal 


Government contemporaneously and under similar terms and conditions; and 


(2) Offers to use the same work force for providing the Federal 
Government with such services as the source uses for providing such services to the 


general public; 


(f) Services of a type offered and sold competitively in substantial quantities 
in the commercial marketplace based on established catalog or market prices for specific 
tasks performed under standard commercial terms and conditions. This does not include 
services that are sold based on hourly rates without an established catalog or market price 


for a specific service performed; 


(g) Any item, combination of items, or service referred to in paragraphs (a) 
through (f), notwithstanding the fact that the item, combination of items, or service is 
transferred between or among separate divisions, subsidiaries, or affiliates of a 


contractor; or 


(h) A nondevelopmental item, if the procuring agency determines the item 
was developed exclusively at private expense and sold in substantial quantities, on a 


competitive basis, to multiple State and local governments. 
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APPENDIX B COMMERCIAL ITEM RISK QUESTIONNAIRE 


The purpose of this questionnaire, which takes about 10 minutes to complete, is to 
identify and investigate the unique risks associated with implementing Commercial-Off- 
The-Shelf (COTS) software application package(s). Answering the questions will help 
you better understand the overall level of risk within your program. It is recommended 
that someone responsible for specifying, procuring and developing software systems 
should complete this questionnaire. After completion, please e-mail the questionnaire to 
rcummins@nps.navy.mil. We believe that you will find the questionnaire both 
interesting and provocative and look forward to receiving your reply. We appreciate your 
help in our research effort, therefore if you would like a copy of our completed study 


please indicate this on the last page of the questionnaire. 
Thank you in advance for your time and co-operation. 
The questionnaire is divided into two parts: 


Section I. Demographic Information. Collecting background information about 


the survey respondents. 


Sections II. Risk questionnaire, a modification to the Information Technology 
Resources Board’s (ITRB) “Risk Profile”, that is organized around three broad 
categories: process, technology, and implementation/logistics support. Each category, 
which represents critical aspects required for the successful implementation of a 


commercial application package(s), is defined below: 


e Process: The key considerations for developing and executing a 
successful acquisition process with the system/program requirements 
driving the organization to consider a commercial solution and the “fit” of 
those requirements with available commercial application package(s). 
Key areas are organizational, planning, tracking, contractual parameters, 
and evaluation of vendor’s experience and past performance. 


° Technology: The technical “fit” of the commercial product(s) with the 
existing and planned technical architecture, which supports an 
organization. This includes the organization’s inherent technical 
challenges, such as the number and complexity of interfaces. 
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e Implementation/Logistics Support: The process contains intermediate 
and final work product characteristics for the delivery of a commercial 
solution within an organization that includes - but is not limited to 
performance measures, vendors availability of support, testing and 
managing organizational change. 


SECTION I 


Service: 





Agency or Organization: 




















Project/System Name: 
Telephone: Fax: 
1. Which category best describes your main job function in your organization? 

a. Management 

b. System analysis or design 

C; Application or system programming 
2, Have you participated in selecting COTS software components that where later 
adapted or integrated into your  project/system? How many _ times? 
3. How long is your work experience with building system from COTS 
components? Years 
4. Please state any good practices, or lessons you have learnt from past experience 


when acquiring and developing systems using COTS software. 














SECTION II 
Questions are organized around the three broad areas of implementing a COTS 
solution as presented above. Each question prompts you, the respondent, to think about 


key factors for a successful COTS application package implementation. You should 
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carefully consider your answer in terms of how it pertains to projects within your own 


organization. 


Answers to each question are provided by the choice a, b or c, which correlate to 


the three levels of risk: low, medium and high, respectively. 


Process 
1. How well are the requirements for your system/program documented? 
a. Thoroughly—comprehensive, current documentation exists 
b. Moderately well—comprehensive documentation exists, but has not been 
updated recently 
Ce Poorly—minimal documentation exists 
2: Because specific requirements are associated with each COTS application 


package(s), how would you describe the relationship between the specifications of the 
COTS product(s) and the requirements of your system/program? 


a. Ideal—great fit, fully meets requirements 
Satisfactory—acceptable fit, meets most requirements 

c. Unsatisfactory—marginal fit, must be modified to meet requirements 
3. How many COTS product(s) can accommodate your system/program 
requirements? 

a. Many 

b. Some 

c. Few 
4. How would you describe the process by which your organization will implement 


new requirements after the initial implementation of the COTS product(s)? 


a. Well-defined, proven process has been established to evaluate and 
implement new requirements (e.g., configuration control board) 
b. Process for evaluating and implementing new requirements has been 
discussed, but not solidified 
C. No process exists for evaluating and implementing new requirements 
De How would you describe your system/program’s ability to adapt to the new 
requirements supported by the COTS product(s)? 


a. Very able—there is a general understanding that the new requirements 
would enhance organization's operation 
b. Somewhat able—there is a general understanding that the new 


requirements would not enhance or deter organization's operation 
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C. Not able—there is a general understanding that the new requirements 
would deter organization's operation 


6. How was the COTS product evaluated and selected? 
a. Well-defined, proven process has been established to evaluate and select 
COTS product 
b. Process for evaluating and selecting COTS products have been discussed, 
but not solidified 
c, Ad hoc, no process exists for evaluating and implementing new 
requirements 

Ai What is the vendor's experience with implementing the COTS product(s) in 


organizations of a size similar to yours? 


a. Extensive experience, established company with quality workforce and 
facilities 

b. Some experience 

C. No experience, company is start-up and situation is highly dynamic 


8. How has the vendor performed in the integration of the COTS application 
package(s) elsewhere? 


a. Excellent past performance 
b. Good past performance 
c. Poor or unknown past performance 
6. What is the vendor's experience with implementing the considered COTS 


product(s) in organizations of a management structure similar to yours? 


a. Extensive experience, established company with quality workforce and 
facilities 

b. Some experience 

C. No experience, company is start-up and situation is highly dynamic 


10. How would you describe the operational control of the organization affected by 
the COTS product(s) implementation? 


a. Centralized 
b. Combination of centralized and decentralized 
Cc. Decentralized 
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11. | How would you describe the sufficiency of skilled staff in the system/program 
affected by the COTS application package(s) implementation? 


a. Sufficiently staffed and skilled 
b. Minimally staffed and skilled 
c. Insufficiently staffed and skilled 


12. | How much experience does the COTS implementation project team have with the 
COTS product(s)? 

a. Extensive experience 

b. Some experience 

C. No experience 
13. | How much experience does the project team have with implementation of other 
COTS products? 

a. Experienced with many COTS products 

b. Experienced with a few COTS products 

C: Experienced with no other COTS products 


14. What is the vendor's track record with implementing the COTS product(s) within 
their cost proposal? 


a. Below total life cycle cost estimate 
b. Met total life cycle cost estimate 
es Exceeded total life cycle cost estimate 
15. How financially stable is the vendor? 
a. Solid financial situation 
b. Mixed financial picture, may have strong revenue but no profit margin 
c. Financial problems, such as poor credit, low revenues and low profit 
margin 
16. To what extent does your acquisition approach include an understanding of the 


vendor's future plans for the COTS product(s)? 


a. Statement of direction for the product, including planned enhancements 
and release dates, has been received 

b. Discussions have been conducted with vendor regarding future direction, 
but no plans have been received in writing 

c No discussion with vendor regarding future direction 
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17. Have data rights been properly negotiated with the vendors? 


a. All data rights negotiated into the contract 
b. Many data rights have been negotiated into the contract 
c. No data rights have been negotiated into the contract 
18. Have cost-effective licensing agreements been worked out with the vendors? 
a. All licensing agreements negotiated into cost 
: Many licensing agreements negotiated into cost 
c Uncertain what licensing agreements are needed 


_ Responses in Process Section: 


fa x1=__ 
| #b x2=_ 
#C_  Xx3=__ 
| Total = 
Technology 
1. Is the COTS application package(s) a totally new system for the organization? 
a. System is a replacement 
b. Components of the system are new 
Cc. New system 
pe To adequately address your organization's needs, what is the level of 


customization required for the COTS product(s) baseline? 


a. No customization necessary 
b. Some customization necessary 
C. Much customization necessary 
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3. How do the COTS application package(s) "fit" with the organization's existing 
and planned architecture? 


a. Good fit 
b. May fit 
c. Not a fit 
4. Is the COTS product(s) view as a time-tested, mature product? 
a. Very mature 
b. Somewhat mature 
C. New or immature 
5 How many functions (e.g., accounting, procurement) are supported by the COTS 


application package(s)? 


a. Single function 
b. Few functions 
C. Many functions 
6. How would you describe the complexity of the interfaces between the COTS 


product(s) and other systems? 


a. Simple, easy to understand 
b. Somewhat complex 
c. Very complex, difficult 


a How many systems interfaces must remain unchanged after the implementation of 
the COTS product(s)? 

a. Few 

b. Some 

C. Many 
8. How would you describe the sufficiency of documentation supporting the 


system(s) with which the COTS application package(s) will interface? 


a. Extremely high quality, thorough documentation 
b. Adequate, some documentation 
C: Poor documentation or does not exist 
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9. To what extent has your organization tested COTS application package(s) in your 
environment? 


a. Conducted extensive testing 
b. Conducted some testing 
C. Have not conducted any testing 


10. Do the security features included in the COTS product(s) need modification to 
meet your organization's requirements? 


a. Meets security requirements, no modification needed 
b. Meets most security requirements, some modification needed 
Cc. Will not handle security requirements, extensive modification needed 


11. How flexible is the design of the COTS product(s) to allow for future changes in 
functionality? 


a. Very flexible—product functions can be easily separated to be modified 
b. Moderately flexible—product functions can be separated to be modified 
G: Not flexible—product functions can not be separated to be modified 


12. What is the reliability history of the COTS product? 


a. Product is stable and has proven itself over time with its customer base 

b. Product has occasional errors but none will result in data loss or other 
critical problem 

c. Product has errors that result in data loss, work lost, system crashes, etc. 


Responses in Technology Section: 


#a x1 
#b x2 
#C x3 


Total = 
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Implementation/Logistics Support 


1. Has your organization examined and applied the lessons learned from other 
organizations that implemented the COTS application package(s)? 


a. Yes—televant lessons learned have been incorporated into the 

implementation plan 

b. Somewhat—past projects have been discussed by the project team 

C. No—have not gathered any information regarding other implementations 
2. How will your organization measure the impact and effectiveness of the COTS 
product(s)? 

a. Comprehensive performance measures (including cost, time spent on each 

activity, etc.) have been established 

b. Performance measures have been discussed but not finalized 

C. No discussion of performance measures 


3 What sort of testing approach is planned for the COTS product(s)? 


a. Designed specifically for a COTS implementation 

b. Combines traditional systems development testing with COTS-specific 
testing 

C. Designed for traditional systems development activities 


4. How would you describe your organization's ability to support new releases of the 
COTS product(s)? 


a. Sufficient—staffing plan for ongoing support of the COTS application 
package(s) has been developed 

b. Moderate—staffing needs have been identified, but plan has not been 
finalized 

C. Minimal—no staff resources are available after the initial implementation 


3. How has the organization prepared for the possibility that the COTS application 
package(s) vendor goes out of business or discontinues support for the product? 


a. Contingency plan finalized and ready to implement 
b. Possibility discussed, but have no finalized plan 
c; Possibility not discussed, no contingency plan being developed 
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6. How would you describe the run time performance of the COTS product(s) in 
your environment? 


a. Very efficient 
b. Moderately efficient 
C. Not efficient 


ve Does the run time performance of the COTS application package(s) meet the 
organization's performance needs? 


a. Efficiently supports the number and location of users 
b. Supports needs with performance degradation 
Cc. Does not support needs 
8. How do other users of the COTS product describe their satisfaction with 


availability of the vendor staff? 


a. Very satisfied, easy to access key personnel at vendor 
b. Somewhat satisfied, can access key personnel some of the time 
c: Unsatisfied, access to key personnel is difficult 
9. What training is needed to operate and maintain the COTS product? 
a. No training 
b. Some training 
CG Extensive training 
a What training sources are available to the customers? 
a. Extensive training resources 
b. Some training resources 
C. No training resources 
11. | How much experience does other support contractors serving your organization in 
functions affected by the COTS implementation have with the COTS application 
package(s)? 
a. Extensive experience 
b. Some experience 
e No experience 


Responses in Implementation/Logistical Support Section: 





APPENDIX C DLA BUSINESS SYSTEMS MODERNIZATION 
QUESTIONNAIRE RESPONSES 








SECTION I 
Service: _ Defense Logistics Agency 
Agency or Organization: __ Defense Logistics Agency 





Project/System Name: Business Systems Modernization 





Which category best describes your main job function in your organization? 


a. Management 
a. System analysis or design 
b. Application or system programming 


Have you participated in selecting COTS software components that where later 
adapted or integrated into your project/system? How many times? First ERP — Other 


minor COTS projects in past — never of this magnitude (enterprise-wide). 


How long is your work experience with building system from COTS 
components? 10 Years 


Please state any good practices, or lessons you have learnt from past experience 
when acquiring and developing systems using COTS software. 


° Do not modify core COTS software 


e Basic integration practices for COTS are the same as_ software 
development (i.e., basics of configuration management, software QA, 
repeatable processes, etc. 


e Willingness to adapt 
° Completeness of requirements 
SECTION II 


Questions are organized around the three broad areas of implementing a COTS 
solution as presented above. Each question prompts you, the respondent, to think about 
key factors for a successful COTS application package implementation. You should 
carefully consider your answer in terms of how it pertains to projects within your own 


organization. 
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Answers to each question are provided by the choice a, b or c, which correlate to 
the three levels of risk: low, medium and high, respectively. Please record your answers 
to the questionnaire directly on this form. We ask that you make the best effort possible 
to provide an answer to all the questions. If you are unsure of an answer, or feel a 
question does not apply to your project, please indicate so rather than leaving a question 
blank. 


Process 
1. How well are the requirements for your system/program documented? 
a. Thoroughly—comprehensive, current documentation exists 
b. Moderately well—comprehensive documentation exists, but has not 
been updated recently 
C; Poorly—minimal documentation exists 
2. Because specific requirements are associated with each COTS application 


package(s), how would you describe the relationship between the specifications of the 


COTS product(s) and the requirements of your system/program? 


a. Ideal—great fit, fully meets requirements 

b. Satisfactory—acceptable fit, meets most requirements 

c. Unsatisfactory—marginal fit, must be modified to meet requirements 
3. How many COTS product(s) can accommodate your system/program 
requirements? 

a. Many 

b. Some 

C. Few 


92 


4. How would you describe the process by which your organization will implement 


new requirements after the initial implementation of the COTS product(s)? 


a. Well-defined, proven process has been established to evaluate and 
implement new requirements (e.g., configuration control board) 

b. Process for evaluating and implementing new requirements has been 
discussed, but not solidified 

Ci No process exists for evaluating and implementing new requirements 
of How would you describe your system/program’s ability to adapt to the new 


requirements supported by the COTS product(s)? 
a. Very able—there is a general understanding that the new 
requirements would enhance organization's operation 


b. Somewhat able—there is a general understanding that the new 
requirements would not enhance or deter organization's operation 


G3 Not able—there is a general understanding that the new requirements 
would deter organization's operation 


6. How was the COTS product evaluated and selected? 

a. Well-defined, proven process has been established to evaluate and 
select cots product 

b. Process for evaluating and selecting COTS products have been discussed, 
but not solidified 

c. Ad hoc, no process exists for evaluating and implementing new 
requirements 
ee What is the vendor's experience with implementing the COTS product(s) in 


organizations of a size similar to yours? 


a. Extensive experience, established company with quality workforce 
and facilities 

b. Some experience 

c. No experience, company is start-up and situation is highly dynamic 
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8. How has the vendor performed in the integration of the COTS application 


package(s) elsewhere? 


a. Excellent past performance 
b. Good past performance 
c. Poor or unknown past performance 
p: What is the vendor's experience with implementing the considered COTS 


product(s) in organizations of a management structure similar to yours? 


a. Extensive experience, established company with quality workforce and 
facilities 
b. Some experience — first major DOD implementation 


C. No experience, company is start-up and situation is highly dynamic 


10. How would you describe the operational control of the organization affected by 


the COTS product(s) implementation? 


a. Centralized 
b. Combination of centralized and decentralized 
Cc. Decentralized 


11. | How would you describe the sufficiency of skilled staff in the system/program 
affected by the COTS application package(s) implementation? 


a. Sufficiently staffed and skilled 
b. Minimally staffed and skilled 
c. Insufficiently staffed and skilled 
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12._ | How much experience does the COTS implementation project team have with the 


COTS product(s)? 


a. Extensive experience 

b. Some experience 

C: No experience 
13. | How much experience does the project team have with implementation of other 
COTS products? 


Experienced with many COTS products 
Experienced with a few COTS products 
Experienced with no other COTS products 


14. What is the vendor's track record with implementing the COTS product(s) within 


their cost proposal? 


a. Below total life cycle cost estimate 
b. Met total life cycle cost estimate 
e: Exceeded total life cycle cost estimate 
15. How financially stable is the vendor? 
a. Solid financial situation 
b. Mixed financial picture, may have strong revenue but no profit margin 
ec. Financial problems, such as poor credit, low revenues and low profit 
margin 
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16. 


To what extent does your acquisition approach include an understanding of the 


vendor's future plans for the COTS product(s)? 


a. Statement of direction for the product, including planned 
enhancements and release dates, has been received 


b. Discussions have been conducted with vendor regarding future direction, 
but no plans have been received in writing 


C No discussion with vendor regarding future direction 





17. Have data rights been properly negotiated with the vendors? 
a. All data rights negotiated into the contract 
b. Many data rights have been negotiated into the contract 
Ci No data rights have been negotiated into the contract 
18. Have cost-effective licensing agreements been worked out with the vendors? 
a. All licensing agreements negotiated into cost 
b. Many licensing agreements negotiated into cost 
¢, Uncertain what licensing agreements are needed 
Responses in Process Section: 
#a12x1= 12 
#b 6 x2= 12 
#c 0x3=0 
Total = 24 
Technology 
1. Is the COTS application package(s) a totally new system for the organization? 


a. System is a replacement 
b. Components of the system are new 
Cc. New system 
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2, To adequately address your organization's needs, what is the level of 


customization required for the COTS product(s) baseline? 


a. No customization necessary 
b. Some customization necessary 
C. Much customization necessary 
J How do the COTS application package(s) "fit" with the organization's existing 


and planned architecture? 


a. Good fit 
b. May fit 
C. Not a fit 
4. Is the COTS product(s) view as a time-tested, mature product? 
a. Very mature 
b. Somewhat mature 
C; New or immature 
3: How many functions (e.g., accounting, procurement) are supported by the COTS 


application package(s)? 


a. Single function 
b. Few functions 
C. Many functions 
6. How would you describe the complexity of the interfaces between the COTS 


product(s) and other systems? 


a. Simple, easy to understand 
b. Somewhat complex 
C. Very complex, difficult 
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re How many systems interfaces must remain unchanged after the implementation of 


the COTS product(s)? 


Few 
b. Some 
C. Many 
8. How would you describe the sufficiency of documentation supporting the 


system(s) with which the COTS application package(s) will interface? 


a. Extremely high quality, thorough documentation 

b. Adequate, some documentation 

c. Poor documentation or does not exist 
9. To what extent has your organization tested COTS application package(s) in your 
environment? 

a. Conducted extensive testing 

b. Conducted some testing 

e: Have not conducted any testing 


10. Do the security features included in the COTS product(s) need modification to 


meet your organization's requirements? 


a. Meets security requirements, no modification needed 
b. Meets most security requirements, some modification needed 
c Will not handle security requirements, extensive modification needed 
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11. How flexible is the design of the COTS product(s) to allow for future changes in 


functionality? 


a. Very flexible—product functions can be easily separated to be modified 


b. Moderately flexible—product functions can be separated to be 
modified 


C; Not flexible—product functions can not be separated to be modified 


12. What is the reliability history of the COTS product? 


a. Product is stable and has proven itself over time with its customer 
base 
b. Product has occasional errors but none will result in data loss or other 


critical problem 


c. Product has errors that result in data loss, work lost, system crashes, etc. 


Responses in Technology Section: 





Implementation/Logistics Support 


1. Has your organization examined and applied the lessons learned from other 


organizations that implemented the COTS application package(s)? 


a. Yes—televant lessons learned have been incorporated into the 
implementation plan 

b. Somewhat—past projects have been discussed by the project team 

c. No—have not gathered any information regarding other implementations 
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2, How will your organization measure the impact and effectiveness of the COTS 


product(s)? 
a. Comprehensive performance measures (including cost, time spent on 
each activity, etc.) have been established 
b. Performance measures have been discussed but not finalized 
on No discussion of performance measures 


ee What sort of testing approach is planned for the COTS product(s)? 


a. Designed specifically for a COTS implementation 


b. Combines traditional systems development testing with COTS-specific 
testing 


Ci Designed for traditional systems development activities 


4. How would you describe your organization's ability to support new releases of the 
COTS product(s)? 
a. Sufficient—staffing plan for ongoing support of the COTS application 
package(s) has been developed 


b. Moderate—staffing needs have been identified, but plan has not been 
finalized 


C. Minimal—no staff resources are available after the initial implementation 


5. How has the organization prepared for the possibility that the COTS application 


package(s) vendor goes out of business or discontinues support for the product? 


a. Contingency plan finalized and ready to implement 
b. Possibility discussed, but have no finalized plan 
ce Possibility not discussed, no contingency plan being developed 
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6. How would you describe the run time performance of the COTS product(s) in 


your environment? 


a. Very efficient 
b. Moderately efficient 
c. Not efficient 
de Does the run time performance of the COTS application package(s) meet the 


organization's performance needs? 


a. Efficiently supports the number and location of users 
b. Supports needs with performance degradation 
Cc. Does not support needs 
8. How do other users of the COTS product describe their satisfaction with 


availability of the vendor staff? 


a. Very satisfied, easy to access key personnel at vendor 
b. Somewhat satisfied, can access key personnel some of the time 
e: Unsatisfied, access to key personnel is difficult 
os What training is needed to operate and maintain the COTS product? 
a. No training 
b. Some training 
C. Extensive training 
10. What training sources are available to the customers? 
a. Extensive training resources 
b. Some training resources 
C. No training resources 
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11. | How much experience does other support contractors serving your organization in 


functions affected by the COTS implementation have with the COTS application 


package(s)? 
a. Extensive experience 
b. Some experience 
(oa No experience 


Responses in Implementation/Logistical Support Section: 
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APPENDIX D ARMY HUMAN RESOURCE SYSTEM 
QUESTIONNAIRE RESPONSES 


SECTION I 


Service: _U.S. Army 


Agency or Organization: PEO EIS, AHRS 
Project/System Name: Army Human Resource System 





Which category best describes your main job function in your organization? 


a. Management 
a. System analysis or design 
b. Application or system programming 


Have you participated in selecting COTS software components that where later 
adapted or integrated into your project/system? How many times? Yes, at least 20. 


How long is your work experience with building system from COTS 
components? 30 Years 


Please state any good practices, or lessons you have learnt from past experience 
when acquiring and developing systems using COTS software. 


e Do not use software that does not have a long-standing commercial user 
base 

° Never be forced to use products with questionable long-term life-cycle 
support 

° Do not allow GOTS products to be forced on your program, these are 


generally built with COTS products no longer in business. 


SECTION II 

Questions are organized around the three broad areas of implementing a COTS 
solution as presented above. Each question prompts you, the respondent, to think about 
key factors for a successful COTS application package implementation. You should 
carefully consider your answer in terms of how it pertains to projects within your own 


organization. 
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Answers to each question are provided by the choice a, b or c, which correlate to 
the three levels of risk: low, medium and high, respectively. Please record your answers 
to the questionnaire directly on this form. We ask that you make the best effort possible 
to provide an answer to all the questions. If you are unsure of an answer, or feel a 
question does not apply to your project, please indicate so rather than leaving a question 
blank. 


Process 
1. How well are the requirements for your system/program documented? 
a. Thoroughly—comprehensive, current documentation exists 
b. Moderately well—comprehensive documentation exists, but has not been 
updated recently 
c Poorly—minimal documentation exists 
2, Because specific requirements are associated with each COTS application 


package(s), how would you describe the relationship between the specifications of the 


COTS product(s) and the requirements of your system/program? 


a. Ideal—great fit, fully meets requirements 

b. Satisfactory—acceptable fit, meets most requirements 

C. Unsatisfactory—marginal fit, must be modified to meet requirements 
3. How many COTS product(s) can accommodate your system/program 
requirements? 

a. Many 

b. Some 

C. Few 
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4. How would you describe the process by which your organization will implement 


new requirements after the initial implementation of the COTS product(s)? 


a. Well-defined, proven process has been established to evaluate and 
implement new requirements (e.g., configuration control board) 

b. Process for evaluating and implementing new requirements has been 
discussed, but not solidified 

Ci No process exists for evaluating and implementing new requirements 
of How would you describe your system/program’s ability to adapt to the new 


requirements supported by the COTS product(s)? 
a. Very able—there is a general understanding that the new 
requirements would enhance organization's operation 


b. Somewhat able—there is a general understanding that the new 
requirements would not enhance or deter organization's operation 


G3 Not able—there is a general understanding that the new requirements 
would deter organization's operation 


6. How was the COTS product evaluated and selected? 

a. Well-defined, proven process has been established to evaluate and 
select cots product 

b. Process for evaluating and selecting COTS products have been discussed, 
but not solidified 

c. Ad hoc, no process exists for evaluating and implementing new 
requirements 
ee What is the vendor's experience with implementing the COTS product(s) in 


organizations of a size similar to yours? 


a. Extensive experience, established company with quality workforce 
and facilities 

b. Some experience 

c. No experience, company is start-up and situation is highly dynamic 
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8. How has the vendor performed in the integration of the COTS application 


package(s) elsewhere? 


a. Excellent past performance 
b. Good past performance 
c. Poor or unknown past performance 
p: What is the vendor's experience with implementing the considered COTS 


product(s) in organizations of a management structure similar to yours? 
a. Extensive experience, established company with quality workforce 
and facilities 
b. Some experience — first major DOD implementation 


c. No experience, company is start-up and situation is highly dynamic 


10. How would you describe the operational control of the organization affected by 


the COTS product(s) implementation? 


a. Centralized 
b. Combination of centralized and decentralized 
Cc. Decentralized 


11. | How would you describe the sufficiency of skilled staff in the system/program 
affected by the COTS application package(s) implementation? 


a. Sufficiently staffed and skilled 
b. Minimally staffed and skilled 
C. Insufficiently staffed and skilled 
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12._ | How much experience does the COTS implementation project team have with the 


COTS product(s)? 


a. Extensive experience 

b. Some experience 

C: No experience 
13. | How much experience does the project team have with implementation of other 
COTS products? 

a. Experienced with many COTS products 


b. Experienced with a few COTS products 
ce Experienced with no other COTS products 


14. What is the vendor's track record with implementing the COTS product(s) within 


their cost proposal? 


a. Below total life cycle cost estimate 
b. Met total life cycle cost estimate 
e: Exceeded total life cycle cost estimate 
15. How financially stable is the vendor? 
a. Solid financial situation 
b. Mixed financial picture, may have strong revenue but no profit margin 
ec. Financial problems, such as poor credit, low revenues and low profit 
margin 
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16. 


To what extent does your acquisition approach include an understanding of the 


vendor's future plans for the COTS product(s)? 


17. 


18. 


a. Statement of direction for the product, including planned 
enhancements and release dates, has been received 


b. Discussions have been conducted with vendor regarding future direction, 
but no plans have been received in writing 


C No discussion with vendor regarding future direction 


Have data rights been properly negotiated with the vendors? 


a. All data rights negotiated into the contract 
b. Many data rights have been negotiated into the contract 
Ci No data rights have been negotiated into the contract 


Have cost-effective licensing agreements been worked out with the vendors? 


a. All licensing agreements negotiated into cost 
b. Many licensing agreements negotiated into cost 
¢, Uncertain what licensing agreements are needed 


Responses in Process Section: 


Pat KT = IT 
#b1x2= 
#c0x3= 


2 
0 


Total = 19 
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Technology 


1. Is the COTS application package(s) a totally new system for the organization? 
a. System is a replacement 
b. Components of the system are new 
Cc. New system 
2: To adequately address your organization's needs, what is the level of 


customization required for the COTS product(s) baseline? 


a. No customization necessary 
b. Some customization necessary 
c. Much customization necessary 
oF How do the COTS application package(s) "fit" with the organization's existing 


and planned architecture? 


a. Good fit 
b. May fit 
Ci Not a fit 
4. Is the COTS product(s) view as a time-tested, mature product? 
a. Very mature 
b. Somewhat mature 
C New or immature 
a How many functions (e.g., accounting, procurement) are supported by the COTS 


application package(s)? 


a. Single function 
b. Few functions 
C. Many functions 
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6. How would you describe the complexity of the interfaces between the COTS 


product(s) and other systems? 


a. Simple, easy to understand 
b. Somewhat complex 


c. Very complex, difficult 


is How many systems interfaces must remain unchanged after the implementation of 
the COTS product(s)? 
Few 
b. Some 
Cc. Many 
8. How would you describe the sufficiency of documentation supporting the 


system(s) with which the COTS application package(s) will interface? 


a. Extremely high quality, thorough documentation 

b. Adequate, some documentation 

e: Poor documentation or does not exist 
9. To what extent has your organization tested COTS application package(s) in your 
environment? 

a. Conducted extensive testing 

b. Conducted some testing 

ic Have not conducted any testing 


110 


10. 


Do the security features included in the COTS product(s) need modification to 


meet your organization's requirements? 


a. Meets security requirements, no modification needed 
b. Meets most security requirements, some modification needed 
c. Will not handle security requirements, extensive modification needed 


11. How flexible is the design of the COTS product(s) to allow for future changes in 
functionality? 
a. Very flexible—product functions can be easily separated to be 
modified 
b. Moderately flexible—product functions can be separated to be modified 
. Not flexible—product functions can not be separated to be modified 
12. What is the reliability history of the COTS product? 


a. Product is stable and has proven itself over time with its customer 
base 
b. Product has occasional errors but none will result in data loss or other 


critical problem 


c. Product has errors that result in data loss, work lost, system crashes, etc. 


Responses in Technology Section: 


#a7x1i=7 
#b 3 x2=6 
#c 2x3=6 


Total = 19 
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Implementation/Logistics Support 


1. Has your organization examined and applied the lessons learned from other 


organizations that implemented the COTS application package(s)? 


a. Yes—relevant lessons learned have been incorporated into the 

implementation plan 

b. Somewhat—past projects have been discussed by the project team 

c. No—have not gathered any information regarding other implementations 
2: How will your organization measure the impact and effectiveness of the COTS 
product(s)? 

a. Comprehensive performance measures (including cost, time spent on 

each activity, etc.) have been established 

b. Performance measures have been discussed but not finalized 

om No discussion of performance measures 


Be What sort of testing approach is planned for the COTS product(s)? 


a. Designed specifically for a COTS implementation 
b. Combines traditional systems development testing with COTS-specific 
testing 
C. Designed for traditional systems development activities 
4. How would you describe your organization's ability to support new releases of the 
COTS product(s)? 
a. Sufficient—been developed 
b. Moderate—staffing needs have been identified, but plan has not been 
finalized 
C. Minimal—no staff resources are available after the initial implementation 
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S: How has the organization prepared for the possibility that the COTS application 


package(s) vendor goes out of business or discontinues support for the product? 


a. Contingency plan finalized and ready to implement 
b. Possibility discussed, but have no finalized plan 
C: Possibility not discussed, no contingency plan being developed 
6. How would you describe the run time performance of the COTS product(s) in 


your environment? 


a. Very efficient 
b. Moderately efficient 
C. Not efficient 
0s Does the run time performance of the COTS application package(s) meet the 


organization's performance needs? 


a. Efficiently supports the number and location of users 
b. Supports needs with performance degradation 
Cc. Does not support needs 
8. How do other users of the COTS product describe their satisfaction with 


availability of the vendor staff? 


a. Very satisfied, easy to access key personnel at vendor 
b. Somewhat satisfied, can access key personnel some of the time 
c, Unsatisfied, access to key personnel is difficult 
9. What training is needed to operate and maintain the COTS product? 
a. No training 
b. Some training 
c Extensive training 
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10. What training sources are available to the customers? 


a. Extensive training resources 
b. Some training resources 
C. No training resources 
11. | How much experience does other support contractors serving your organization in 


functions affected by the COTS implementation have with the COTS application 


package(s)? 
a. Extensive experience 
b. Some experience 
e: No experience 
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APPENDIX E ARMY COMMUNICATION SOFTWARE SUPPORT 
DIVIDION QUESTIONNAIRE RESPONSES 


SECTION I 


Service: U.S. Army 








Agency or Organization: CECOM-SEC or AMSEL-SE-WS-COM 
Project/System Name: Post Production Software Support 





Which category best describes your main job function in your organization? 


a. Management and Software Support 
b. System analysis or design 
C. Application or system programming 


Have you participated in selecting COTS software components that where later 
adapted or integrated into your project/system? How many times? We made 


recommendations based on the legality use and when COTS products are no longer 
supported _or reach _end_of_ life. There were two incidents where our 
recommendations of COTS replacement were integrated. 


How long is your work experience with building system from COTS 
components? 4Years 


Please state any good practices, or lessons you have learnt from past experience 
when acquiring and developing systems using COTS software. 
° Know your requirements well 


e Assess and evaluate different available COTS products and its cost based 
on the requirements way in advance 


° Close, continuous, and active partnership among the vendor, customers, 
developer, and most importantly the users 


SECTION I 


Questions are organized around the three broad areas of implementing a COTS 
solution as presented above. Each question prompts you, the respondent, to think about 
key factors for a successful COTS application package implementation. You should 
carefully consider your answer in terms of how it pertains to projects within your own 


organization. 
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Answers to each question are provided by the choice a, b or c, which correlate to 
the three levels of risk: low, medium and high, respectively. Please record your answers 
to the questionnaire directly on this form. We ask that you make the best effort possible 
to provide an answer to all the questions. If you are unsure of an answer, or feel a 
question does not apply to your project, please indicate so rather than leaving a question 


blank. 


Process 
1. How well are the requirements for your system/program documented? 
a. Thoroughly—comprehensive, current documentation exists 
b. Moderately well—comprehensive documentation exists, but has not been 
updated recently 
c Poorly—minimal documentation exists 
2, Because specific requirements are associated with each COTS application 


package(s), how would you describe the relationship between the specifications of the 


COTS product(s) and the requirements of your system/program? 


a. Ideal—great fit, fully meets requirements 

b. Satisfactory—acceptable fit, meets most requirements 

C. Unsatisfactory—marginal fit, must be modified to meet requirements 
3. How many COTS product(s) can accommodate your system/program 
requirements? 

a. Many 

b. Some 

C. Few 
4. How would you describe the process by which your organization will implement 


new requirements after the initial implementation of the COTS product(s)? 
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a. Well-defined, proven process has been established to evaluate and 
implement new requirements (e.g., configuration control board) 


b. Process for evaluating and implementing new requirements has been 
discussed, but not solidified 


c. No process exists for evaluating and implementing new requirements 


». How would you describe your system/program’s ability to adapt to the new 
requirements supported by the COTS product(s)? 
a. Very able—there is a general understanding that the new 
requirements would enhance organization's operation 


b. Somewhat able—there is a general understanding that the new 
requirements would not enhance or deter organization's operation 


c. Not able—there is a general understanding that the new requirements 
would deter organization's operation 


6. How was the COTS product evaluated and selected? 
a. Well-defined, proven process has been established to evaluate and select 
cots product 
b. Process for evaluating and selecting COTS products have been 
discussed, but not solidified 
c. Ad hoc, no process exists for evaluating and implementing new 
requirements 

fa What is the vendor's experience with implementing the COTS product(s) in 


organizations of a size similar to yours? 


a. Extensive experience, established company with quality workforce and 
facilities 

b. Some experience 

c. No experience, company is start-up and situation is highly dynamic 


117 


8. How has the vendor performed in the integration of the COTS application 


package(s) elsewhere? 


a. Excellent past performance 
b. Good past performance 
C. Poor or unknown past performance 
p: What is the vendor's experience with implementing the considered COTS 


product(s) in organizations of a management structure similar to yours? 
a. Extensive experience, established company with quality workforce and 
facilities 
b. Some experience 


C. No experience, company is start-up and situation is highly dynamic 


10. | How would you describe the operational control of the organization affected by 


the COTS product(s) implementation? 


a. Centralized 
b. Combination of centralized and decentralized 
Cc. Decentralized 


11. | How would you describe the sufficiency of skilled staff in the system/program 
affected by the COTS application package(s) implementation? 


a. Sufficiently staffed and skilled 
b. Minimally staffed and skilled 
C. Insufficiently staffed and skilled 
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12._ | How much experience does the COTS implementation project team have with the 


COTS product(s)? 


a. Extensive experience 

b. Some experience 

C: No experience 
13. | How much experience does the project team have with implementation of other 
COTS products? 

a. Experienced with many COTS products 


b. Experienced with a few COTS products 
c Experienced with no other COTS products 


14. What is the vendor's track record with implementing the COTS product(s) within 


their cost proposal? 


a. Below total life cycle cost estimate 
b. Met total life cycle cost estimate 
C. Exceeded total life cycle cost estimate 
15. How financially stable is the vendor? 
a. Solid financial situation 
b. Mixed financial picture, may have strong revenue but no profit 
margin 
ee Financial problems, such as poor credit, low revenues and low profit 
margin 
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16. 


To what extent does your acquisition approach include an understanding of the 


vendor's future plans for the COTS product(s)? 


17. 


18. 


a. Statement of direction for the product, including planned enhancements 
and release dates, has been received 


b. Discussions have been conducted with vendor regarding future direction, 
but no plans have been received in writing 


C No discussion with vendor regarding future direction 


Have data rights been properly negotiated with the vendors? 


a. All data rights negotiated into the contract 
b. Many data rights have been negotiated into the contract 
Ci No data rights have been negotiated into the contract 


Have cost-effective licensing agreements been worked out with the vendors? 


a. All licensing agreements negotiated into cost 
b. Many licensing agreements negotiated into cost 
¢, Uncertain what licensing agreements are needed 


Responses in Process Section: 
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Technology 


1. Is the COTS application package(s) a totally new system for the organization? 
a. System is a replacement 
b. Components of the system are new 
Cc. New system 
Z To adequately address your organization's needs, what is the level of 


customization required for the COTS product(s) baseline? 


a. No customization necessary 
b. Some customization necessary 
es Much customization necessary 
oF How do the COTS application package(s) "fit" with the organization's existing 


and planned architecture? 


a. Good fit 
b. May fit 
Ci Nota fit 
4. Is the COTS product(s) view as a time-tested, mature product? 
a. Very mature 
b. Somewhat mature 
C. New or immature 
a: How many functions (e.g., accounting, procurement) are supported by the COTS 


application package(s)? 


a. Single function 
b. Few functions 
C. Many functions 
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6. How would you describe the complexity of the interfaces between the COTS 


product(s) and other systems? 


a. Simple, easy to understand 

b. Somewhat complex 

C. Very complex, difficult 
is How many systems interfaces must remain unchanged after the implementation of 
the COTS product(s)? 

Few 

b. Some 

Cc. Many 
8. How would you describe the sufficiency of documentation supporting the 


system(s) with which the COTS application package(s) will interface? 


a. Extremely high quality, thorough documentation 

b. Adequate, some documentation 

c. Poor documentation or does not exist 
9. To what extent has your organization tested COTS application package(s) in your 
environment? 

a. Conducted extensive testing 

b. Conducted some testing 

ic Have not conducted any testing 
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10. _ Do the security features included in the COTS product(s) need modification to 


meet your organization's requirements? 


a. Meets security requirements, no modification needed 
b. Meets most security requirements, some modification needed 
c. Will not handle security requirements, extensive modification needed 


11. How flexible is the design of the COTS product(s) to allow for future changes in 


functionality? 
a. Very flexible—product functions can be easily separated to be modified 
b. Moderately flexible—product functions can be separated to be modified 
C. Not flexible—product functions can not be separated to be modified 


12. What is the reliability history of the COTS product? 


a. Product is stable and has proven itself over time with its customer base 

b. Product has occasional errors but none will result in data loss or other 
critical problem 

C Product has errors that result in data loss, work lost, system crashes, etc. 


Responses in Technology Section: 


#a2x1=2 
#b 7x2= 14 


#co 3x3=9 


Total = 
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Implementation/Logistics Support 


1. Has your organization examined and applied the lessons learned from other 


organizations that implemented the COTS application package(s)? 


a. Yes—televant lessons learned have been incorporated into the 
implementation plan 
b. Somewhat—past projects have been discussed by the project team 
Cc. No—have not gathered any information § regarding other 
implementations 
2s How will your organization measure the impact and effectiveness of the COTS 
product(s)? 
a. Comprehensive performance measures (including cost, time spent on each 
activity, etc.) have been established 
b. Performance measures have been discussed but not finalized 
oe No discussion of performance measures 


3; What sort of testing approach is planned for the COTS product(s)? 


a. Designed specifically for a COTS implementation 


b. Combines traditional systems development testing with COTS-specific 
testing 


c. Designed for traditional systems development activities 


4. How would you describe your organization's ability to support new releases of the 


COTS product(s)? 


a. Sufficient—staffing plan for ongoing support of the COTS application 
package(s) has been developed 


b. Moderate—staffing needs have been identified, but plan has not been 
finalized 
c. Minimal—no staff resources are available after the initial implementation 
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S: How has the organization prepared for the possibility that the COTS application 


package(s) vendor goes out of business or discontinues support for the product? 


a. Contingency plan finalized and ready to implement 
b. Possibility discussed, but have no finalized plan 
C. Possibility not discussed, no contingency plan being developed 
6. How would you describe the run time performance of the COTS product(s) in 


your environment? 


a. Very efficient 
b. Moderately efficient 
C. Not efficient 


7 Does the run time performance of the COTS application package(s) meet the 


organization's performance needs? 


a. Efficiently supports the number and location of users 
b. Supports needs with performance degradation 
Cc. Does not support needs 
8. How do other users of the COTS product describe their satisfaction with 


availability of the vendor staff? 


a. Very satisfied, easy to access key personnel at vendor 
b. Somewhat satisfied, can access key personnel some of the time 
c Unsatisfied, access to key personnel is difficult 
9: What training is needed to operate and maintain the COTS product? 
a. No training 
b. Some training 
C. Extensive training 
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10. What training sources are available to the customers? 


a. Extensive training resources 
b. Some training resources 
C. No training resources 
11. | How much experience does other support contractors serving your organization in 


functions affected by the COTS implementation have with the COTS application 


package(s)? 
a. Extensive experience 
b. Some experience 
e: No experience 


Responses in Implementation/Logistical Support Section: 
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APPENDIX F ARMY GLOBAL COMBAT SUPPORT SYSTEM 
QUESTIONNAIRE RESPONSES 


SECTION I 


Service: _ U.S. Army 


Agency or Organization: __ PM Logistics Information Systems 
Project/System Name: Global Combat Support System — Army Tactical 





Which category best describes your main job function in your organization? 


a. Management 
b. System analysis or design 
C. Application or system programming 


Have you participated in selecting COTS software components that where later 
adapted or integrated into your project/system? How many times? Yes, several 


How long is your work experience with building system from COTS 
components? 1Years 


Please state any good practices, or lessons you have learnt from past experience 
when acquiring and developing systems using COTS software. 


SECTION I 


Questions are organized around the three broad areas of implementing a COTS 
solution as presented above. Each question prompts you, the respondent, to think about 
key factors for a successful COTS application package implementation. You should 
carefully consider your answer in terms of how it pertains to projects within your own 
organization. 

Answers to each question are provided by the choice a, b or c, which correlate to 
the three levels of risk: low, medium and high, respectively. Please record your answers 
to the questionnaire directly on this form. We ask that you make the best effort possible 
to provide an answer to all the questions. If you are unsure of an answer, or feel a 
question does not apply to your project, please indicate so rather than leaving a question 


blank. 
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Process 


1. How well are the requirements for your system/program documented? 


a. Thoroughly—comprehensive, current documentation exists 


b. Moderately well—comprehensive documentation exists, but has not been 
updated recently 


C; Poorly—minimal documentation exists 


2. Because specific requirements are associated with each COTS application 
package(s), how would you describe the relationship between the specifications of the 


COTS product(s) and the requirements of your system/program? 


a. Ideal—great fit, fully meets requirements 

b. Satisfactory—acceptable fit, meets most requirements 

c. Unsatisfactory—marginal fit, must be modified to meet requirements 
3. How many COTS product(s) can accommodate your system/program 
requirements? 

a. Many 

b. Some 

c. Few 
4. How would you describe the process by which your organization will implement 


new requirements after the initial implementation of the COTS product(s)? 
a. Well-defined, proven process has been established to evaluate and 
implement new requirements (e.g., configuration control board) 


b. Process for evaluating and implementing new requirements has been 
discussed, but not solidified 


C. No process exists for evaluating and implementing new requirements 
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5. How would you describe your system/program’s ability to adapt to the new 
requirements supported by the COTS product(s)? 
a. Very able—there is a general understanding that the new 
requirements would enhance organization's operation 


b. Somewhat able—there is a general understanding that the new 
requirements would not enhance or deter organization's operation 


CG; Not able—there is a general understanding that the new requirements 
would deter organization's operation 


6. How was the COTS product evaluated and selected? 
a. Well-defined, proven process has been established to evaluate and 
select cots product 
b. Process for evaluating and selecting COTS products have been discussed, 
but not solidified 
c. Ad hoc, no process exists for evaluating and implementing new 
requirements 

ae What is the vendor's experience with implementing the COTS product(s) in 


organizations of a size similar to yours? 
a. Extensive experience, established company with quality workforce 
and facilities 
b. Some experience 


C. No experience, company is start-up and situation is highly dynamic 


8. How has the vendor performed in the integration of the COTS application 


package(s) elsewhere? 


a. Excellent past performance 
b. Good past performance 
C. Poor or unknown past performance 
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9. What is the vendor's experience with implementing the considered COTS 
product(s) in organizations of a management structure similar to yours? 
a. Extensive experience, established company with quality workforce 
and facilities 
b. Some experience — first major DOD implementation 


Ci No experience, company is start-up and situation is highly dynamic 


10. | How would you describe the operational control of the organization affected by 


the COTS product(s) implementation? 


a. Centralized 
b. Combination of centralized and decentralized 
Cc. Decentralized 


11. | How would you describe the sufficiency of skilled staff in the system/program 
affected by the COTS application package(s) implementation? 


a. Sufficiently staffed and skilled 
b. Minimally staffed and skilled 
c. Insufficiently staffed and skilled 


12. How much experience does the COTS implementation project team have with the 
COTS product(s)? 

a. Extensive experience 

b. Some experience 

c. No experience 
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13. | How much experience does the project team have with implementation of other 


COTS products? 


a. Experienced with many COTS products 
b. Experienced with a few COTS products 
C. Experienced with no other COTS products 


14. What is the vendor's track record with implementing the COTS product(s) within 


their cost proposal? (Impossible to answer) 


a. Below total life cycle cost estimate 
b. Met total life cycle cost estimate 
c Exceeded total life cycle cost estimate 
15. How financially stable is the vendor? 
a. Solid financial situation 
b. Mixed financial picture, may have strong revenue but no profit margin 
C; Financial problems, such as poor credit, low revenues and low profit 
margin 
16. To what extent does your acquisition approach include an understanding of the 


vendor's future plans for the COTS product(s)? 
a. Statement of direction for the product, including planned 
enhancements and release dates, has been received 


b. Discussions have been conducted with vendor regarding future direction, 
but no plans have been received in writing 


c. No discussion with vendor regarding future direction 


17. Have data rights been properly negotiated with the vendors? 


a. All data rights negotiated into the contract 
b. Many data rights have been negotiated into the contract 
c. No data rights have been negotiated into the contract 
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18. Have cost-effective licensing agreements been worked out with the vendors? 


a. All licensing agreements negotiated into cost 
b. Many licensing agreements negotiated into cost 
C. Uncertain what licensing agreements are needed 


Responses in Process Section: 





Technology 
1. Is the COTS application package(s) a totally new system for the organization? 
a. System is a replacement 
b. Components of the system are new 
Cc. New system 
2. To adequately address your organization's needs, what is the level of 


customization required for the COTS product(s) baseline? 


a. No customization necessary 
b. Some customization necessary 
C, Much customization necessary 
3: How do the COTS application package(s) "fit" with the organization's existing 


and planned architecture? 


a. Good fit 
b. May fit 
Cc. Not a fit 
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4. Is the COTS product(s) view as a time-tested, mature product? 


a. Very mature 
b. Somewhat mature 
(oF New or immature 
ef How many functions (e.g., accounting, procurement) are supported by the COTS 


application package(s)? 


a. Single function 
b. Few functions 
c. Many functions 
6. How would you describe the complexity of the interfaces between the COTS 


product(s) and other systems? 


a. Simple, easy to understand 
b. Somewhat complex 


c. Very complex, difficult 


7. How many systems interfaces must remain unchanged after the implementation of 
the COTS product(s)? 
Few 
b. Some 
Cc. Many 
8. How would you describe the sufficiency of documentation supporting the 


system(s) with which the COTS application package(s) will interface? 


a. Extremely high quality, thorough documentation 
b. Adequate, some documentation 
(oH Poor documentation or does not exist 
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9. To what extent has your organization tested COTS application package(s) in your 


environment? 
a. Conducted extensive testing 
b. Conducted some testing 
C. Have not conducted any testing 


10. Do the security features included in the COTS product(s) need modification to 


meet your organization's requirements? 


a. Meets security requirements, no modification needed 
b. Meets most security requirements, some modification needed 
c Will not handle security requirements, extensive modification needed 


11. | How flexible is the design of the COTS product(s) to allow for future changes in 


functionality? 


a. Very flexible—product functions can be easily separated to be 
modified 


b. Moderately flexible—product functions can be separated to be modified 


G; Not flexible—product functions can not be separated to be modified 


12. What is the reliability history of the COTS product? 


a. Product is stable and has proven itself over time with its customer 
base 

b. Product has occasional errors but none will result in data loss or other 
critical problem 

c. Product has errors that result in data loss, work lost, system crashes, etc. 


Responses in Technology Section: 
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Implementation/Logistics Support 


1. Has your organization examined and applied the lessons learned from other 


organizations that implemented the COTS application package(s)? 


a. Yes—relevant lessons learned have been incorporated into the 

implementation plan 

b. Somewhat—past projects have been discussed by the project team 

c. No—have not gathered any information regarding other implementations 
2: How will your organization measure the impact and effectiveness of the COTS 
product(s)? 

a. Comprehensive performance measures (including cost, time spent on 

each activity, etc.) have been established 

b. Performance measures have been discussed but not finalized 

c No discussion of performance measures 


2. What sort of testing approach is planned for the COTS product(s)? 


a. Designed specifically for a COTS implementation 
b. Combines traditional systems development testing with COTS-specific 
testing 
c. Designed for traditional systems development activities 
4. How would you describe your organization's ability to support new releases of the 
COTS product(s)? 
a. Sufficient—staffing plan for ongoing support of the COTS application 
package(s) has been developed 
b. Moderate—staffing needs have been identified, but plan has not been 
finalized 
C. Minimal—no staff resources are available after the initial implementation 
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S: How has the organization prepared for the possibility that the COTS application 


package(s) vendor goes out of business or discontinues support for the product? 


a. Contingency plan finalized and ready to implement 
b. Possibility discussed, but have no finalized plan 
C Possibility not discussed, no contingency plan being developed 
6. How would you describe the run time performance of the COTS product(s) in 


your environment? 


a. Very efficient 
b. Moderately efficient 
c. Not efficient 
7 Does the run time performance of the COTS application package(s) meet the 


organization's performance needs? 


a. Efficiently supports the number and location of users 
b. Supports needs with performance degradation 
Cc. Does not support needs 
8. How do other users of the COTS product describe their satisfaction with 


availability of the vendor staff? 


a. Very satisfied, easy to access key personnel at vendor 
b. Somewhat satisfied, can access key personnel some of the time 
e Unsatisfied, access to key personnel is difficult 
9: What training is needed to operate and maintain the COTS product? 
a. No training 
b. Some training 
ce Extensive training 
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10. What training sources are available to the customers? 


a. Extensive training resources 
b. Some training resources 
C. No training resources 
11. | How much experience does other support contractors serving your organization in 


functions affected by the COTS implementation have with the COTS application 


package(s)? 
a. Extensive experience 
b. Some experience 
e: No experience 


Responses in Implementation/Logistical Support Section: 


#a 10 x1= 10 
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APPENDIX G MARINE CORPS COMBAT VEHICLE TRAINING 
SIMULATOR QUESTIONNAIRE RESPONSES 


SECTION I 


Service: __ United States Marine Corps 








Agency or Organization: Marine Corps Systems Command 
Project/System Name: Global Transportation and Engineer Systems 





Which category best describes your main job function in your organization? 


a. Management 
b. System analysis or design 
C. Application or system programming 


Have you participated in selecting COTS software components that where later 
adapted or integrated into your project/system? How many times? No. 


How long is your work experience with building system from COTS 
components? 5Years 


Please state any good practices, or lessons you have learnt from past experience 
when acquiring and developing systems using COTS software. 

° Combined Synopsis/Solicitation are great tools 
SECTION II 

Questions are organized around the three broad areas of implementing a COTS 
solution as presented above. Each question prompts you, the respondent, to think about 
key factors for a successful COTS application package implementation. You should 
carefully consider your answer in terms of how it pertains to projects within your own 
organization. 

Answers to each question are provided by the choice a, b or c, which correlate to 
the three levels of risk: low, medium and high, respectively. Please record your answers 
to the questionnaire directly on this form. We ask that you make the best effort possible 
to provide an answer to all the questions. If you are unsure of an answer, or feel a 
question does not apply to your project, please indicate so rather than leaving a question 


blank. 
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Process 


1. How well are the requirements for your system/program documented? 


a. Thoroughly—comprehensive, current documentation exists 


b. Moderately well—comprehensive documentation exists, but has not 
been updated recently 


Cy Poorly—minimal documentation exists 


2. Because specific requirements are associated with each COTS application 
package(s), how would you describe the relationship between the specifications of the 


COTS product(s) and the requirements of your system/program? 


a. Ideal—great fit, fully meets requirements 

b. Satisfactory—acceptable fit, meets most requirements 

¢, Unsatisfactory—amarginal fit, must be modified to meet requirements 
3. How many COTS product(s) can accommodate your system/program 
requirements? 

a. Many 

b. Some 

c. Few 
4. How would you describe the process by which your organization will implement 


new requirements after the initial implementation of the COTS product(s)? 
a. Well-defined, proven process has been established to evaluate and 
implement new requirements (e.g., configuration control board) 


b. Process for evaluating and implementing new requirements has been 
discussed, but not solidified 


C. No process exists for evaluating and implementing new requirements 
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5. How would you describe your system/program’s ability to adapt to the new 
requirements supported by the COTS product(s)? 
a. Very able—there is a general understanding that the new 
requirements would enhance organization's operation 


b. Somewhat able—there is a general understanding that the new 
requirements would not enhance or deter organization's operation 


CG; Not able—there is a general understanding that the new requirements 
would deter organization's operation 


6. How was the COTS product evaluated and selected? 
a. Well-defined, proven process has been established to evaluate and select 
cots product 
b. Process for evaluating and selecting COTS products have been 
discussed, but not solidified 
c. Ad hoc, no process exists for evaluating and implementing new 
requirements 

ae What is the vendor's experience with implementing the COTS product(s) in 


organizations of a size similar to yours? 
a. Extensive experience, established company with quality workforce 
and facilities 
b. Some experience 


C. No experience, company is start-up and situation is highly dynamic 


8. How has the vendor performed in the integration of the COTS application 


package(s) elsewhere? 


a. Excellent past performance 
b. Good past performance 
C. Poor or unknown past performance 
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9. What is the vendor's experience with implementing the considered COTS 
product(s) in organizations of a management structure similar to yours? 
a. Extensive experience, established company with quality workforce and 
facilities 
b. Some experience — first major DOD implementation 


Ci No experience, company is start-up and situation is highly dynamic 


10. | How would you describe the operational control of the organization affected by 


the COTS product(s) implementation? 


a. Centralized 
b. Combination of centralized and decentralized 
Cc. Decentralized 


11. | How would you describe the sufficiency of skilled staff in the system/program 
affected by the COTS application package(s) implementation? 


a. Sufficiently staffed and skilled 
b. Minimally staffed and skilled 
c. Insufficiently staffed and skilled 


12._ | How much experience does the COTS implementation project team have with the 
COTS product(s)? 

a. Extensive experience 

b. Some experience 

c. No experience 
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13. | How much experience does the project team have with implementation of other 


COTS products? 


a. Experienced with many COTS products 
b. Experienced with a few COTS products 
C. Experienced with no other COTS products 


14. What is the vendor's track record with implementing the COTS product(s) within 


their cost proposal? 


a. Below total life cycle cost estimate 
b. Met total life cycle cost estimate 
c. Exceeded total life cycle cost estimate 
15. How financially stable is the vendor? 
a. Solid financial situation 
b. Mixed financial picture, may have strong revenue but no profit 
margin 
os Financial problems, such as poor credit, low revenues and low profit 
margin 
16. To what extent does your acquisition approach include an understanding of the 


vendor's future plans for the COTS product(s)? 
a. Statement of direction for the product, including planned enhancements 
and release dates, has been received 


b. Discussions have been conducted with vendor regarding future 
direction, but no plans have been received in writing 


Cc, No discussion with vendor regarding future direction 


17. _Have data rights been properly negotiated with the vendors? 


a. All data rights negotiated into the contract 
b. Many data rights have been negotiated into the contract 
c. No data rights have been negotiated into the contract 
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18. Have cost-effective licensing agreements been worked out with the vendors? 


a. All licensing agreements negotiated into cost 
b. Many licensing agreements negotiated into cost 
C. Uncertain what licensing agreements are needed 


Responses in Process Section: 





Technology 
1. Is the COTS application package(s) a totally new system for the organization? 
a. System is a replacement 
b. Components of the system are new 
Cc. New system 
2. To adequately address your organization's needs, what is the level of 


customization required for the COTS product(s) baseline? 


a. No customization necessary 
b. Some customization necessary 
C. Much customization necessary 
3: How do the COTS application package(s) "fit" with the organization's existing 


and planned architecture? 


a. Good fit 
b. May fit 
Cc. Not a fit 
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4. Is the COTS product(s) view as a time-tested, mature product? 


a. Very mature 
b. Somewhat mature 
C. New or immature 
ef How many functions (e.g., accounting, procurement) are supported by the COTS 


application package(s)? 


a. Single function 
b. Few functions 
C. Many functions 
6. How would you describe the complexity of the interfaces between the COTS 


product(s) and other systems? 


a. Simple, easy to understand 
b. Somewhat complex 


c. Very complex, difficult 


7. How many systems interfaces must remain unchanged after the implementation of 
the COTS product(s)? 
Few 
b. Some 
Cc. Many 
8. How would you describe the sufficiency of documentation supporting the 


system(s) with which the COTS application package(s) will interface? 


a. Extremely high quality, thorough documentation 
b. Adequate, some documentation 
(oH Poor documentation or does not exist 
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9. To what extent has your organization tested COTS application package(s) in your 


environment? 
a. Conducted extensive testing 
b. Conducted some testing 
C. Have not conducted any testing 


10. Do the security features included in the COTS product(s) need modification to 


meet your organization's requirements? 


a. Meets security requirements, no modification needed 
b. Meets most security requirements, some modification needed 
c Will not handle security requirements, extensive modification needed 


11. | How flexible is the design of the COTS product(s) to allow for future changes in 


functionality? 


a. Very flexible—product functions can be easily separated to be 
modified 


b. Moderately flexible—product functions can be separated to be modified 


G; Not flexible—product functions can not be separated to be modified 


12. What is the reliability history of the COTS product? 


a. Product is stable and has proven itself over time with its customer 
base 

b. Product has occasional errors but none will result in data loss or other 
critical problem 

c. Product has errors that result in data loss, work lost, system crashes, etc. 


Responses in Technology Section: 


ao) 
21 
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Implementation/Logistics Support 


1. Has your organization examined and applied the lessons learned from other 


organizations that implemented the COTS application package(s)? 


a. Yes—televant lessons learned have been incorporated into the 
implementation plan 
b. Somewhat—past projects have been discussed by the project team 
Cc. No—have not gathered any information’ regarding other 
implementations 
2s How will your organization measure the impact and effectiveness of the COTS 
product(s)? 
a. Comprehensive performance measures (including cost, time spent on each 
activity, etc.) have been established 
b. Performance measures have been discussed but not finalized 
oe No discussion of performance measures 


3; What sort of testing approach is planned for the COTS product(s)? 


a. Designed specifically for a COTS implementation 


b. Combines traditional systems development testing with COTS-specific 
testing 


C. Designed for traditional systems development activities 


4. How would you describe your organization's ability to support new releases of the 


COTS product(s)? 


a. Sufficient—staffing plan for ongoing support of the COTS application 
package(s) has been developed 


b. Moderate—staffing needs have been identified, but plan has not been 
finalized 
c. Minimal—no staff resources are available after the initial implementation 
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S: How has the organization prepared for the possibility that the COTS application 


package(s) vendor goes out of business or discontinues support for the product? 


a. Contingency plan finalized and ready to implement 
b. Possibility discussed, but have no finalized plan 
C: Possibility not discussed, no contingency plan being developed 
6. How would you describe the run time performance of the COTS product(s) in 


your environment? 


a. Very efficient 
b. Moderately efficient 
C. Not efficient 
0s Does the run time performance of the COTS application package(s) meet the 


organization's performance needs? 


a. Efficiently supports the number and location of users 
b. Supports needs with performance degradation 
Cc. Does not support needs 
8. How do other users of the COTS product describe their satisfaction with 


availability of the vendor staff? 


a. Very satisfied, easy to access key personnel at vendor 
b. Somewhat satisfied, can access key personnel some of the time 
c Unsatisfied, access to key personnel is difficult 
9: What training is needed to operate and maintain the COTS product? 
a. No training 
b. Some training 
C. Extensive training 
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10. What training sources are available to the customers? 


a. Extensive training resources 
b. Some training resources 
Cc. No training resources (very little) 
11. | How much experience does other support contractors serving your organization in 


functions affected by the COTS implementation have with the COTS application 


package(s)? 
a. Extensive experience 
b. Some experience 
e: No experience 


Responses in Implementation/Logistical Support Section: 
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APPENDIX H ARMY COMMON SOFTWARE PROGRAM 
QUESTIONNAIRE RESPONSES 


SECTION I 


Service: _ U.S. Army 


Agency or Organization: _ PM Ground Combat Command & Control (GC C2) 
Project/System Name: Common Software 





Which category best describes your main job function in your organization? 


a. Management 
b. System analysis or design 
C. Application or system programming 


Have you participated in selecting COTS software components that where later 
adapted or integrated into your project/system? How many times? Yes, one time. 


How long is your work experience with building system from COTS 
components? 2Years 


Please state any good practices, or lessons you have learnt from past experience 
when acquiring and developing systems using COTS software. 


e Never rely on a single vendor for critical functionality, always have 
alternate products lined up 


e Carefully consider the likelihood that the vendor will not be there to 
support it in the future 


SECTION I 


Questions are organized around the three broad areas of implementing a COTS 
solution as presented above. Each question prompts you, the respondent, to think about 
key factors for a successful COTS application package implementation. You should 
carefully consider your answer in terms of how it pertains to projects within your own 
organization. 

Answers to each question are provided by the choice a, b or c, which correlate to 
the three levels of risk: low, medium and high, respectively. Please record your answers 
to the questionnaire directly on this form. We ask that you make the best effort possible 


to provide an answer to all the questions. If you are unsure of an answer, or feel a 
151 


question does not apply to your project, please indicate so rather than leaving a question 


blank. 


Process 
k, How well are the requirements for your system/program documented? 
a. Thoroughly—comprehensive, current documentation exists 
b. Moderately well—comprehensive documentation exists, but has not 
been updated recently 
C; Poorly—minimal documentation exists 
2. Because specific requirements are associated with each COTS application 


package(s), how would you describe the relationship between the specifications of the 


COTS product(s) and the requirements of your system/program? 


a. Ideal—great fit, fully meets requirements 

b. Satisfactory—acceptable fit, meets most requirements 

C. Unsatisfactory—amarginal fit, must be modified to meet requirements 
3. How many COTS product(s) can accommodate your system/program 
requirements? 

a. Many 

b. Some 

c. Few 
4. How would you describe the process by which your organization will implement 


new requirements after the initial implementation of the COTS product(s)? 
a. Well-defined, proven process has been established to evaluate and 
implement new requirements (e.g., configuration control board) 


b. Process for evaluating and implementing new requirements has been 
discussed, but not solidified 


C. No process exists for evaluating and implementing new requirements 
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5. How would you describe your system/program’s ability to adapt to the new 
requirements supported by the COTS product(s)? 
a. Very able—there is a general understanding that the new requirements 
would enhance organization's operation 


b. Somewhat able—there is a general understanding that the new 
requirements would not enhance or deter organization's operation 


CG: Not able—there is a general understanding that the new requirements 
would deter organization's operation 


6. How was the COTS product evaluated and selected? 
a. Well-defined, proven process has been established to evaluate and select 
cots product 
b. Process for evaluating and selecting COTS products have been 
discussed, but not solidified 
c. Ad hoc, no process exists for evaluating and implementing new 
requirements 

ae What is the vendor's experience with implementing the COTS product(s) in 


organizations of a size similar to yours? 
a. Extensive experience, established company with quality workforce 
and facilities 
b. Some experience 


C. No experience, company is start-up and situation is highly dynamic 


8. How has the vendor performed in the integration of the COTS application 


package(s) elsewhere? 


a. Excellent past performance 
b. Good past performance 
C. Poor or unknown past performance 
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9. What is the vendor's experience with implementing the considered COTS 
product(s) in organizations of a management structure similar to yours? 
a. Extensive experience, established company with quality workforce 
and facilities 
b. Some experience — first major DOD implementation 


Ci No experience, company is start-up and situation is highly dynamic 


10. | How would you describe the operational control of the organization affected by 


the COTS product(s) implementation? 


a. Centralized 
b. Combination of centralized and decentralized 
Cc. Decentralized 


11. | How would you describe the sufficiency of skilled staff in the system/program 
affected by the COTS application package(s) implementation? 


a. Sufficiently staffed and skilled 
b. Minimally staffed and skilled 
c. Insufficiently staffed and skilled 


12._ | How much experience does the COTS implementation project team have with the 
COTS product(s)? 

a. Extensive experience 

b. Some experience 

c No experience 
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13. | How much experience does the project team have with implementation of other 


COTS products? 


a. Experienced with many COTS products 
b. Experienced with a few COTS products 
C. Experienced with no other COTS products 


14. What is the vendor's track record with implementing the COTS product(s) within 


their cost proposal? (Impossible to answer) 


a. Below total life cycle cost estimate 
b. Met total life cycle cost estimate 
c. Exceeded total life cycle cost estimate 
15. How financially stable is the vendor? 
a. Solid financial situation 
b. Mixed financial picture, may have strong revenue but no profit margin 
C; Financial problems, such as poor credit, low revenues and low profit 
margin 
16. To what extent does your acquisition approach include an understanding of the 


vendor's future plans for the COTS product(s)? 
a. Statement of direction for the product, including planned 
enhancements and release dates, has been received 


b. Discussions have been conducted with vendor regarding future direction, 
but no plans have been received in writing 


c. No discussion with vendor regarding future direction 


17. Have data rights been properly negotiated with the vendors? 


a. All data rights negotiated into the contract 
b. Many data rights have been negotiated into the contract 
c. No data rights have been negotiated into the contract 
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18. Have cost-effective licensing agreements been worked out with the vendors? 


a. All licensing agreements negotiated into cost 
b. Many licensing agreements negotiated into cost 
C. Uncertain what licensing agreements are needed 


Responses in Process Section: 





Technology 
1. Is the COTS application package(s) a totally new system for the organization? 
a. System is a replacement 
b. Components of the system are new 
Cc. New system 
2. To adequately address your organization's needs, what is the level of 


customization required for the COTS product(s) baseline? 


a. No customization necessary 
b. Some customization necessary 
C, Much customization necessary 
3: How do the COTS application package(s) "fit" with the organization's existing 


and planned architecture? 


a. Good fit 
b. May fit 
Cc. Not a fit 
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4. Is the COTS product(s) view as a time-tested, mature product? 


a. Very mature 
b. Somewhat mature 
Cc. New or immature 
ef How many functions (e.g., accounting, procurement) are supported by the COTS 


application package(s)? 


a. Single function 
b. Few functions 
C. Many functions 
6. How would you describe the complexity of the interfaces between the COTS 


product(s) and other systems? 


a. Simple, easy to understand 
b. Somewhat complex 


c. Very complex, difficult 


7. How many systems interfaces must remain unchanged after the implementation of 
the COTS product(s)? 
Few 
b. Some 
Cc. Many 
8. How would you describe the sufficiency of documentation supporting the 


system(s) with which the COTS application package(s) will interface? 


a. Extremely high quality, thorough documentation 
b. Adequate, some documentation 
(oH Poor documentation or does not exist 
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9. To what extent has your organization tested COTS application package(s) in your 


environment? 
a. Conducted extensive testing 
b. Conducted some testing 
C. Have not conducted any testing 


10. Do the security features included in the COTS product(s) need modification to 


meet your organization's requirements? 


a. Meets security requirements, no modification needed 
b. Meets most security requirements, some modification needed 
c Will not handle security requirements, extensive modification needed 


11. | How flexible is the design of the COTS product(s) to allow for future changes in 


functionality? 


a. Very flexible—product functions can be easily separated to be 
modified 


b. Moderately flexible—product functions can be separated to be modified 


G; Not flexible—product functions can not be separated to be modified 


12. What is the reliability history of the COTS product? 


a. Product is stable and has proven itself over time with its customer base 

b. Product has occasional errors but none will result in data loss or other 
critical problem 

c. Product has errors that result in data loss, work lost, system crashes, etc. 


Responses in Technology Section: 
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Implementation/Logistics Support 


1. Has your organization examined and applied the lessons learned from other 


organizations that implemented the COTS application package(s)? 


a. Yes—televant lessons learned have been incorporated into the 
implementation plan 
b. Somewhat—past projects have been discussed by the project team 
Cc. No—have not gathered any information’ regarding other 
implementations 
2s How will your organization measure the impact and effectiveness of the COTS 
product(s)? 
a. Comprehensive performance measures (including cost, time spent on each 
activity, etc.) have been established 
b. Performance measures have been discussed but not finalized 
oe No discussion of performance measures 


3; What sort of testing approach is planned for the COTS product(s)? 


a. Designed specifically for a COTS implementation 
b. Combines traditional systems development testing with COTS-specific 
testing 
C. Designed for traditional systems development activities 
4. How would you describe your organization's ability to support new releases of the 
COTS product(s)? 
a. Sufficient—staffing plan for ongoing support of the COTS application 
package(s) has been developed 
b. Moderate—staffing needs have been identified, but plan has not been 
finalized 
C. Minimal—no staff resources are available after the initial implementation 
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S: How has the organization prepared for the possibility that the COTS application 


package(s) vendor goes out of business or discontinues support for the product? 


a. Contingency plan finalized and ready to implement 
b. Possibility discussed, but have no finalized plan 
C. Possibility not discussed, no contingency plan being developed 
6. How would you describe the run time performance of the COTS product(s) in 


your environment? 


a. Very efficient 
b. Moderately efficient 
c. Not efficient 
0s Does the run time performance of the COTS application package(s) meet the 


organization's performance needs? 


a. Efficiently supports the number and location of users 
b. Supports needs with performance degradation 
Cc. Does not support needs 
8. How do other users of the COTS product describe their satisfaction with 


availability of the vendor staff? 


a. Very satisfied, easy to access key personnel at vendor 
b. Somewhat satisfied, can access key personnel some of the time 
c, Unsatisfied, access to key personnel is difficult 
9. What training is needed to operate and maintain the COTS product? 
a. No training 
b. Some training 
em Extensive training 
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10. What training sources are available to the customers? 


a. Extensive training resources 
b. Some training resources 
C. No training resources 
11. | How much experience does other support contractors serving your organization in 


functions affected by the COTS implementation have with the COTS application 


package(s)? 
a. Extensive experience 
b. Some experience 
C. No experience 


Responses in Implementation/Logistical Support Section: 
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